Allison Vacanti
1819b003c2
Update test json files with new number format and multi-device.
2021-07-08 14:44:18 -04:00
Allison Vacanti
861f66c161
Update comparison script.
...
- Handle multiple benchmarks.
- Split output to resemble nvbench markdown output:
- Headings for benchmarks / devices
- Compact tables with axis values / stats
- Handle `null` stdev (these are infinite, e.g. too few samples)
- Format numbers, times, percentages similar to nvbench.
- Add summary of total/pass/unknown/failure stats
- Add new test.json files (with multiple benchmarks in each)
2021-06-22 16:15:22 -04:00
Vyas Ramasubramani
f1715786fd
Add bracketing newlines.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
1002082817
Apply black formatting.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
dae1f16426
Add some intermeidate variables and simplify code to make it self-documenting.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
053eb493c7
Add nonzero error code on exit if any tests fail and change output format to Github-style Markdown.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ab39c924b2
Add comment regarding better metrics of failure.
2021-06-22 12:30:05 -04:00
Ashwin Srinath
0f63f86e9f
Report pass/fail status
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
38ec66dc0f
Add small comment.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
5db95906f8
Convert named values to a key-value mapping from a list.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
c2868dfa35
Convert states into a key-value mapping.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
0491e9c17f
Store axes and summaries as dictionaries rather than lists.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
8fcad4908f
Improve table formatting, add calculated performance difference.
2021-06-22 12:30:05 -04:00
Ashwin Srinath
d9704f0db1
Add pretty printing of table
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ed4feade63
Use f-string with explicit format width.
2021-06-22 12:30:05 -04:00
Allison Vacanti
1fc4576ceb
Initial draft of python comparison script.
2021-06-22 12:29:53 -04:00
