NVIDIA/nvbench
1
0
Fork 0
mirror of https://github.com/NVIDIA/nvbench.git synced 2026-04-20 14:58:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
102 Commits 8 Branches 5 Tags
bb871094c32659538bb7c9b04f56ac1626ed14f8
Commit Graph

46 Commits

Author SHA1 Message Date
Allison Vacanti
bb871094c3 Fixes for multidevice/gcc. 
- Allow devices to be cleared during benchmark definition.
- Fix various demangling bugs.
 2021-02-15 21:26:21 -05:00
Allison Vacanti
6c67578dcd Implement skip_time and improve logging. 2021-02-15 17:39:46 -05:00
Allison Vacanti
d323f569b8 Add termination criteria API. 
- min_samples
- min_time
- max_noise
- skip_time (not yet implemented)
- timeout

Refactored s/(trials)|(iters)/samples/s.
 2021-02-15 12:04:15 -05:00
Allison Vacanti
92cc3b1189 Execute benchmarks on all devices. 2021-02-12 20:53:10 -05:00
Allison Vacanti
9f9c6e5278 Refactor state_generator. 
The old implementation was scattered and ad hoc. This one is slightly
less so.

More importantly, refactoring to this design will make it easier to
add device traversal.
 2021-02-11 21:24:58 -05:00
Allison Vacanti
56d182ad41 Fix float64_axis test. 
Changed to use `{:0.5g}` formatting for input strings until I figure
out something better.
 2021-02-11 21:13:48 -05:00
Allison Vacanti
3bc8291b28 Allow benchmarks to be specified by index with --benchmark. 2021-02-10 21:40:39 -05:00
Allison Vacanti
0477514bb6 Axis spec revamp. 
- Add support for single values ("Axis=Value").
- Make other value specs shell friendly:
  - Range: "Axis:(2:10:2)"  -> "Axis=[2:10:2]"
  - List:  "Axis:{2,3,4,5}" -> "Axis=[2,3,4,5]"
  - ":" -> "=" feels more natural
  - "{}()" characters have special meaning in bash.
  - "[]" character don't require escapes.
  - Using the same braces for both ranges/list is easier to remember,
    only the delimiter changes.
 2021-02-10 09:55:50 -05:00
Allison Vacanti
4bdc27de8c More fixes for int64_t != long long. 2021-02-04 19:02:58 -05:00
Allison Vacanti
74d19a8e16 Clean up formatting. 2021-02-04 18:54:44 -05:00
Allison Vacanti
e302583c67 Fix usages of ASSERT_MSG that creates empty __VA_ARGS__. 
Invoking a variadic macro with zero variadic args is illegal until
C++20. The extra calls to fmt::format were unnecessary, anyway.
 2021-02-04 18:54:20 -05:00
Allison Vacanti
6bb22b952c Add option parsing to NVBENCH_MAIN. 
- Convert benchmark_manager into a read-only structure.
  - Mutable benchmarks will be provided by
    `option_parser::get_benchmarks()` or
    `benchmark_manager::clone_benchmarks()`.
 2021-02-04 13:15:21 -05:00
Allison Vacanti
efd4442d1b Add option_parser. 
Currently supports `--benchmark` and `--axis` options.
 2021-02-03 21:39:17 -05:00
Allison Vacanti
b48da531b0 Implement/test mutable string_axis. 2021-02-03 13:37:01 -05:00
Allison Vacanti
e732e50120 Implement/test mutable int64_axis. 2021-02-03 13:32:19 -05:00
Allison Vacanti
36de02d2be Implement/test mutable float64_axis. 2021-02-03 12:50:50 -05:00
Allison Vacanti
6aac1554d5 Add type axis masking. 
Allow types in a type_axis to be masked out, skipping all
benchmark configurations that use a masked out type.

This will allow the axis to be modified at runtime with CLI args.

Requires various changes to support mutable axes.
 2021-02-02 18:24:26 -05:00
Allison Vacanti
6dd4bebe6e Clonable benchmarks. 
Adds clone() methods to benchmark_base and axis_base, along with the
scaffolding to support them.
 2021-01-14 17:51:13 -05:00
Allison Vacanti
0f3f1ef899 Polishing up the proof of concept. 
- Enable throughput stats
- Add cold measurements
- Print liveness/log messages while running trials.
- Improve markdown formatting
- nvbench::range
 2021-01-02 01:45:12 -05:00
Allison Vacanti
93ed09f2b3 Finish off proof of concept level functionality. 
Add `measure_hot` and `markdown_format` as detail classes, since
they're still pretty rough and need to be worked into the design.
But for now, they're functional proofs of concept. The formatting is
rough, but demonstrates basics.

Also added launch, exec, and summary functionality. These aren't details
and will be part of the project's API.
 2021-01-01 21:47:20 -05:00
Allison Vacanti
3ffd2e6aea Add NVBENCH_CREATE and associated machinery. 2021-01-01 01:36:53 -05:00
Allison Vacanti
0776bdc4be Add nvbench::runner. 2020-12-31 21:27:51 -05:00
Allison Vacanti
29d7eafc33 cartesian_product<type_list<...>> -- now with edge cases! 
Add test cases for zero or one input type_list.
 2020-12-31 19:51:18 -05:00
Allison Vacanti
b1cbf6cab8 Add test for typeless state_generator::create. 
Turns out this edgecase was already handled correctly, but let's
make sure it stays that way.
 2020-12-31 17:07:19 -05:00
Allison Vacanti
a1bd7b03b3 Update state_generator::create to handle type axes. 2020-12-31 16:28:41 -05:00
Allison Vacanti
fd3b82cc6f Move sleep kernel to its own header. 
This will be useful for testing in general.
 2020-12-30 14:47:25 -05:00
Allison Vacanti
ad44463d6e Replace params class with nvbench::named_values. 
Refactor nvbench::state to use this for axis parameters.

These will also be useful for summaries and measurements.

Also adds a new ASSERT_THROWS_ANY macro to test some of the new API.
 2020-12-30 14:45:46 -05:00
Allison Vacanti
8c0b8e3423 Add cpu_timer. 2020-12-29 23:51:09 -05:00
Allison Vacanti
b07ffafff4 Add cuda_timer, cuda_stream. 2020-12-29 23:50:39 -05:00
Allison Vacanti
981031e732 Update axes_metadata test to reflect new API. 2020-12-29 19:41:29 -05:00
Allison Vacanti
e631f1ff03 Add state_generator::create method to encapsulate state creation. 2020-12-29 19:35:07 -05:00
Allison Vacanti
beaead2c3f Split benchmark into more specialized, nontemplated structs. 2020-12-29 19:34:11 -05:00
Allison Vacanti
093077de5f Add nvbench::state. 
This class holds a single value for each runtime axis.
 2020-12-27 10:44:22 -05:00
Allison Vacanti
7b14ceb3fe Add detail::state_generator. 
This helper utility computes the cartesian product of the runtime
axes.
 2020-12-27 10:29:24 -05:00
Allison Vacanti
40f92b4705 Add initial nvbench::benchmark. 
It's basically just a container for the various axis classes at this
point.
 2020-12-24 17:33:03 -05:00
Allison Vacanti
691ed2c18d Add nvbench::params. 2020-12-22 17:44:33 -05:00
Allison Vacanti
76f9c9b0d6 Add nvbench::string_axis. 2020-12-22 16:57:42 -05:00
Allison Vacanti
1e5fe88c9b Add float64_axis. 2020-12-22 16:38:05 -05:00
Allison Vacanti
65eadda1c1 Add type_axis::get_index. 2020-12-22 16:37:19 -05:00
Allison Vacanti
07e4cc36c2 Add type_axis. 2020-12-22 15:20:04 -05:00
Allison Vacanti
95e2eaf607 Add nvbench::tl::foreach. 2020-12-22 15:20:04 -05:00
Allison Vacanti
fb10be7e72 Add nvbench::tl::size. 2020-12-22 15:20:04 -05:00
Allison Vacanti
1c04e49787 Rename type_list header to match convention. 2020-12-22 15:20:03 -05:00
Allison Vacanti
334178dfa8 Tweak int64_axis API. 
- Add `int64_axis_flags`.
- Rename `axis_base` virtuals.
 2020-12-22 15:20:03 -05:00
Allison Vacanti
13dc404d56 Add int64_axis. 2020-12-21 20:31:12 -05:00
Allison Vacanti
014d94e402 Add nvbench::type_list. 2020-12-20 21:09:47 -05:00