Robert Maynard
35dd8de2ce
Remove unneeded scripts/requirements.txt
2021-12-20 13:24:24 -05:00
Robert Maynard
f9b44378bf
nvbench_compare now supports comparing directories of results
2021-12-16 16:26:13 -05:00
Robert Maynard
905f84272e
Add --threshold-diff command option to nvbench_compare
...
Allows us to filter output to only see the significantly different
benchmarks
2021-12-16 15:52:30 -05:00
Robert Maynard
52d9aed8da
refactor to have a proper main entry point
2021-12-16 15:27:51 -05:00
Robert Maynard
3f6d496824
Add a requirements.txt for the nv_bench script
2021-12-16 13:44:40 -05:00
Allison Vacanti
9054880881
Support older versions of tabulate module.
...
Ubuntu 18.04 only provides tabulate v0.7, but colalign and github
formatting options were added in 0.8.3.
2021-07-08 14:48:02 -04:00
Allison Vacanti
a32f2d7eed
Handle benchmarks with no axes in compare script.
2021-07-08 14:46:54 -04:00
Allison Vacanti
ab91baf5d8
Fix number parsing in compare script.
...
All numbers are now stored as strings to avoid truncation of 64-bit ints.
2021-07-08 14:45:27 -04:00
Allison Vacanti
1819b003c2
Update test json files with new number format and multi-device.
2021-07-08 14:44:18 -04:00
Allison Vacanti
861f66c161
Update comparison script.
...
- Handle multiple benchmarks.
- Split output to resemble nvbench markdown output:
- Headings for benchmarks / devices
- Compact tables with axis values / stats
- Handle `null` stdev (these are infinite, e.g. too few samples)
- Format numbers, times, percentages similar to nvbench.
- Add summary of total/pass/unknown/failure stats
- Add new test.json files (with multiple benchmarks in each)
2021-06-22 16:15:22 -04:00
Vyas Ramasubramani
f1715786fd
Add bracketing newlines.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
1002082817
Apply black formatting.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
dae1f16426
Add some intermeidate variables and simplify code to make it self-documenting.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
053eb493c7
Add nonzero error code on exit if any tests fail and change output format to Github-style Markdown.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ab39c924b2
Add comment regarding better metrics of failure.
2021-06-22 12:30:05 -04:00
Ashwin Srinath
0f63f86e9f
Report pass/fail status
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
38ec66dc0f
Add small comment.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
5db95906f8
Convert named values to a key-value mapping from a list.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
c2868dfa35
Convert states into a key-value mapping.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
0491e9c17f
Store axes and summaries as dictionaries rather than lists.
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
8fcad4908f
Improve table formatting, add calculated performance difference.
2021-06-22 12:30:05 -04:00
Ashwin Srinath
d9704f0db1
Add pretty printing of table
2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ed4feade63
Use f-string with explicit format width.
2021-06-22 12:30:05 -04:00
Allison Vacanti
1fc4576ceb
Initial draft of python comparison script.
2021-06-22 12:29:53 -04:00
