NVIDIA/nvbench
1
0
Fork 0
mirror of https://github.com/NVIDIA/nvbench.git synced 2026-03-14 20:27:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
571 Commits 10 Branches 4 Tags
pull-request/80
Commit Graph

37 Commits

Author SHA1 Message Date
Allison Piper
3440855dbd Formatting updates. 2025-04-14 17:26:12 +00:00
Bernhard Manfred Gruber
f52aa4b0aa Distinguish slower, same and faster comparisons (#190) 
Fixes: #178
 2024-11-15 12:41:47 -05:00
Georgii Evtushenko
0ce45af043 Plot comparison results (#90) 2024-11-13 14:28:04 -05:00
Georgy Evtushenko
1eb0c49222 Adopt histogram script to new format 2022-12-22 17:12:22 +04:00
Allison Vacanti
8ba2cf1395 Update compare script and test files for new JSON. 2022-01-11 17:55:36 -05:00
Allison Vacanti
9995eb67fd Add JSON file version checks to python scripts. 2022-01-11 15:06:54 -05:00
Georgy Evtushenko
61afb8d7e7 Initial implementation of nvbench_histogram. 2022-01-11 15:06:54 -05:00
Allison Vacanti
f1c985955a Clean up JSON output for consistency and easier parsing. 
- Prefer an array of objects with `.name` fields over key/value pairs
  for arbitrary collections of objects.
- Write common summary value names directly as fields.
 2022-01-11 15:06:54 -05:00
Allison Vacanti
b11c0ba3a0 Add a binary JSON output (--jsonbin) that dumps timing samples. 2022-01-11 15:06:54 -05:00
Allison Vacanti
74e96e8618 Add nvbench_walltime.py script. 2022-01-11 15:06:54 -05:00
Allison Vacanti
6dee1eec3b Refactor summary API and update nvbench/summary.cuh docs. 
The string used when constructing a summary is no longer a human
readable name, but rather a tag string (e.g. "nv/cold/time/gpu/mean").
These will make lookup easier and more stable going forward.

name vs. short_name no longer exists. Now there is just "name", which
is used for column headings. The "description" string may still be
used for detailed information.

Updated the json tests and compare script to reflect these changes.
 2022-01-11 15:06:26 -05:00
Allison Vacanti
3c01814945 Skip non-json files and empty files in compare script. 2021-12-21 21:03:02 -05:00
Robert Maynard
6c1f372c45 Allow nvbench [-flags] (files|dirs) 2021-12-20 13:31:32 -05:00
Robert Maynard
35dd8de2ce Remove unneeded scripts/requirements.txt 2021-12-20 13:24:24 -05:00
Robert Maynard
f9b44378bf nvbench_compare now supports comparing directories of results 2021-12-16 16:26:13 -05:00
Robert Maynard
905f84272e Add --threshold-diff command option to nvbench_compare 
Allows us to filter output to only see the significantly different
benchmarks
 2021-12-16 15:52:30 -05:00
Robert Maynard
52d9aed8da refactor to have a proper main entry point 2021-12-16 15:27:51 -05:00
Robert Maynard
3f6d496824 Add a requirements.txt for the nv_bench script 2021-12-16 13:44:40 -05:00
Allison Vacanti
9054880881 Support older versions of tabulate module. 
Ubuntu 18.04 only provides tabulate v0.7, but colalign and github
formatting options were added in 0.8.3.
 2021-07-08 14:48:02 -04:00
Allison Vacanti
a32f2d7eed Handle benchmarks with no axes in compare script. 2021-07-08 14:46:54 -04:00
Allison Vacanti
ab91baf5d8 Fix number parsing in compare script. 
All numbers are now stored as strings to avoid truncation of 64-bit ints.
 2021-07-08 14:45:27 -04:00
Allison Vacanti
1819b003c2 Update test json files with new number format and multi-device. 2021-07-08 14:44:18 -04:00
Allison Vacanti
861f66c161 Update comparison script. 
- Handle multiple benchmarks.
- Split output to resemble nvbench markdown output:
  - Headings for benchmarks / devices
  - Compact tables with axis values / stats
- Handle `null` stdev (these are infinite, e.g. too few samples)
- Format numbers, times, percentages similar to nvbench.
- Add summary of total/pass/unknown/failure stats
- Add new test.json files (with multiple benchmarks in each)
 2021-06-22 16:15:22 -04:00
Vyas Ramasubramani
f1715786fd Add bracketing newlines. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
1002082817 Apply black formatting. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
dae1f16426 Add some intermeidate variables and simplify code to make it self-documenting. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
053eb493c7 Add nonzero error code on exit if any tests fail and change output format to Github-style Markdown. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ab39c924b2 Add comment regarding better metrics of failure. 2021-06-22 12:30:05 -04:00
Ashwin Srinath
0f63f86e9f Report pass/fail status 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
38ec66dc0f Add small comment. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
5db95906f8 Convert named values to a key-value mapping from a list. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
c2868dfa35 Convert states into a key-value mapping. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
0491e9c17f Store axes and summaries as dictionaries rather than lists. 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
8fcad4908f Improve table formatting, add calculated performance difference. 2021-06-22 12:30:05 -04:00
Ashwin Srinath
d9704f0db1 Add pretty printing of table 2021-06-22 12:30:05 -04:00
Vyas Ramasubramani
ed4feade63 Use f-string with explicit format width. 2021-06-22 12:30:05 -04:00
Allison Vacanti
1fc4576ceb Initial draft of python comparison script. 2021-06-22 12:29:53 -04:00