Markdown Report

[IlyasMoutawwakil]

Making reporting much better with grouped benchmark report logging and printing.
The benchmark report class BenchmarkReport can now be viewed with .log() or .print(). It can also be transformed entirely into text, through to_plain_text and to_markdown_text.
The benchmark config class BenchmarkConfig get two additional arguments print_report (default false) and log_report (default true) to control this behavior.

new report logging:

[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - + load:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ memory:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ max_ram: 949.44 (MB)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ latency:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ count: 1
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ total: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ mean: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p50: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p90: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p95: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p99: 1.244811 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ stdev: 0.000000 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ stdev_: 0.00 (%)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - + forward:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ memory:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ max_ram: 960.98 (MB)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ latency:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ count: 22
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ total: 1.023933 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ mean: 0.046542 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p50: 0.046848 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p90: 0.049224 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p95: 0.049346 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ p99: 0.049694 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ stdev: 0.002056 (s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ stdev_: 4.42 (%)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ throughput:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ throughput: 21.49 (samples/s)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ energy:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ cpu: 0.000002 (kWh)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ gpu: 0.000000 (kWh)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ ram: 0.000000 (kWh)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ total: 0.000002 (kWh)
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 	+ efficiency:
[PYTEST-PROCESS][2024-10-03 11:52:30][benchmark][INFO] - 		+ efficiency: 574475.856681 (samples/kWh)

new report printing:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                    load:                                     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                    memory:                                     
                           
  metric     value   unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━ 
  max_ram   949.44     MB  
                           
                                    latency:                                    
                            
  metric      value   unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  count           1      -  
  total    1.244811      s  
  mean     1.244811      s  
  p50      1.244811      s  
  p90      1.244811      s  
  p95      1.244811      s  
  p99      1.244811      s  
  stdev    0.000000      s  
  stdev_       0.00      %  
                            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                   forward:                                   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                    memory:                                     
                           
  metric     value   unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━ 
  max_ram   960.98     MB  
                           
                                    latency:                                    
                            
  metric      value   unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  count          22      -  
  total    1.023933      s  
  mean     0.046542      s  
  p50      0.046848      s  
  p90      0.049224      s  
  p95      0.049346      s  
  p99      0.049694      s  
  stdev    0.002056      s  
  stdev_       4.42      %  
                            
                                  throughput:                                   
                                  
  metric       value        unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  throughput   21.49   samples/s  
                                  
                                    energy:                                     
                            
  metric      value   unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  cpu      0.000002    kWh  
  gpu      0.000000    kWh  
  ram      0.000000    kWh  
  total    0.000002    kWh  
                            
                                  efficiency:                                   
                                            
  metric               value          unit  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  efficiency   574475.856681   samples/kWh  

[baptistecolle]

Hey, great PR. It would be cool and, I believe, quite quick to generate a printed report after a benchmark to a txt file.

More or less when you run optimum-benchmark --config-dir examples/ --config-name pytorch_bert
you generate:

- pytorch_bert/
  - .hydra/
  - benchmark_config.json
  - benchmark_report.json
  - benchmark.json
  - benchmark.log
  - cli.log

It would be nice to add to this folder structure
benchmark_report.txt with the output of to_plain_text or benchmark_report.md with to_markdown_text

Because the jsons are not very human readable, so having a md or txt output after a benchmark could be a nice quality-of-life update