Commit Graph

60 Commits

Author SHA1 Message Date
Allison Vacanti
4e83e048ba Store percentages as ratios.
Human-readable outputs (md) and CLI inputs still use percentages.
In-memory and machine-readable outputs (csv, json) use ratios.

This is the convention that spreadsheet apps expect. Fixes #2.
2021-03-18 13:42:43 -04:00
Allison Vacanti
ea53972af8 Add nvbench.all metatarget.
This builds all NVBench tests and examples without building targets in
any parent projects.
2021-03-18 13:33:23 -04:00
Allison Vacanti
ab8fc13aac Simplify enum_type_list usage.
Using C++17 auto template parameters enables two improvements:

1. Enum type no longer needed in `enum_type_list`:

```
// Old:
nvbench::enum_type_list<my_enum, my_enum::value1, my_enum::value2, ...>
// New:
nvbench::enum_type_list<my_enum::value1, my_enum::value2, ...>
```

2. New `nvbench::enum_type` helper cuts down on verbosity when writing
benchmarks

```
// Old
template <typename T, my_enum_a MyEnumA, my_enum_b MyEnumB>
void my_bench(nvbench::state &state,
              nvbench::type_list<T,
                                 std::integral_constant<my_enum_a,
                                                        MyEnumA>,
                                 std::integral_constant<my_enum_b,
                                                        MyEnumB>>)
{ ... }

// New
template <typename T, my_enum_a MyEnumA, my_enum_b MyEnumB>
void my_bench(nvbench::state &state,
              nvbench::type_list<T,
                                 nvbench::enum_type<MyEnumA>,
                                 nvbench::enum_type<MyEnumB>>)
{ ... }
```
2021-03-18 02:59:51 -04:00
Allison Vacanti
9f6404bac6 Pad range max for floating point types.
This avoids an annoying case where the max value is dropped due to
rounding errors.

Adds a few other missing test cases for `nvbench::range`, too.

Fixes #3.
2021-03-16 15:12:53 -04:00
Allison Vacanti
56414078ea Added state::get_XXXX_or_default(axis_name, default_value) 2021-03-16 14:36:20 -04:00
Allison Vacanti
60c94d9ed6 Add enum_type_axis and examples/enums.cu.
- `enum_type_axis` simplifies using integral_constants with type axes.
- `examples/enums.cu` demonstrates various ways of implementing parameter
  sweeps with enum types.
2021-03-16 13:57:52 -04:00
Allison Vacanti
f15b668b03 Add nvbench.test.all and nvbench.example.all metatargets. 2021-03-09 16:03:14 -05:00
Allison Vacanti
cc01d8863c Fix copyright year. 2021-03-08 18:39:27 -05:00
Allison Vacanti
c133784763 Add the first example.
- New NVBench_ENABLE_EXAMPLES CMake option.
- examples/axis.cu provides examples of parameter sweeps.
- Moves testing/sleep_kernel.cuh -> nvbench/test_kernels.cuh
  - Accessible to examples and provides some built-in kernels for users
    to experiement with.
  - Not included with `<nvbench/nvbench.cuh>`.
2021-03-08 18:26:26 -05:00
Allison Vacanti
078919b6b4 Add LICENSE file and boilerplate.
for i in `find  ./ -regex ".*\.cuh?$"`
do
  sh -c "cat ~/boilerplate ${i} > tmp && mv tmp ${i}"
done
2021-03-04 16:03:02 -05:00
Allison Vacanti
cf71f6ee15 Update NVBench build system with initial standalone support. 2021-03-03 13:59:29 -05:00
Allison Vacanti
8d6d934dfe Add default axis names.
Also cleaned up the annoying quirk where `set_type_axes_names` *had*
to be called on all benchmarks with type axes.

Default names are {"T", "U", "V", "W"} for up-to four type axes. For
five or more, {"T0", "T1", ...} is used instead.
2021-02-19 12:37:05 -05:00
Allison Vacanti
a747982415 Add nvbench::main CMake target.
Linking to this instead of `nvbench::nvbench` will automatically include
the `NVBENCH_MAIN` macro.
2021-02-19 09:34:02 -05:00
Allison Vacanti
37e753f7b6 Update benchmark macros:
s/NVBENCH_CREATE/NVBENCH_BENCH/g
s/NVBENCH_BENCH_TEMPLATE/NVBENCH_BENCH_TYPES/g

This will fit nicer once the exec_tags version are added:

NVBENCH_BENCH
NVBENCH_BENCH_TYPES
NVBENCH_BENCH_FLAGS
NVBENCH_BENCH_TYPES_FLAGS
2021-02-16 16:08:38 -05:00
Allison Vacanti
bb871094c3 Fixes for multidevice/gcc.
- Allow devices to be cleared during benchmark definition.
- Fix various demangling bugs.
2021-02-15 21:26:21 -05:00
Allison Vacanti
6c67578dcd Implement skip_time and improve logging. 2021-02-15 17:39:46 -05:00
Allison Vacanti
d323f569b8 Add termination criteria API.
- min_samples
- min_time
- max_noise
- skip_time (not yet implemented)
- timeout

Refactored s/(trials)|(iters)/samples/s.
2021-02-15 12:04:15 -05:00
Allison Vacanti
92cc3b1189 Execute benchmarks on all devices. 2021-02-12 20:53:10 -05:00
Allison Vacanti
9f9c6e5278 Refactor state_generator.
The old implementation was scattered and ad hoc. This one is slightly
less so.

More importantly, refactoring to this design will make it easier to
add device traversal.
2021-02-11 21:24:58 -05:00
Allison Vacanti
56d182ad41 Fix float64_axis test.
Changed to use `{:0.5g}` formatting for input strings until I figure
out something better.
2021-02-11 21:13:48 -05:00
Allison Vacanti
3bc8291b28 Allow benchmarks to be specified by index with --benchmark. 2021-02-10 21:40:39 -05:00
Allison Vacanti
0477514bb6 Axis spec revamp.
- Add support for single values ("Axis=Value").
- Make other value specs shell friendly:
  - Range: "Axis:(2:10:2)"  -> "Axis=[2:10:2]"
  - List:  "Axis:{2,3,4,5}" -> "Axis=[2,3,4,5]"
  - ":" -> "=" feels more natural
  - "{}()" characters have special meaning in bash.
  - "[]" character don't require escapes.
  - Using the same braces for both ranges/list is easier to remember,
    only the delimiter changes.
2021-02-10 09:55:50 -05:00
Allison Vacanti
4bdc27de8c More fixes for int64_t != long long. 2021-02-04 19:02:58 -05:00
Allison Vacanti
74d19a8e16 Clean up formatting. 2021-02-04 18:54:44 -05:00
Allison Vacanti
e302583c67 Fix usages of ASSERT_MSG that creates empty __VA_ARGS__.
Invoking a variadic macro with zero variadic args is illegal until
C++20. The extra calls to fmt::format were unnecessary, anyway.
2021-02-04 18:54:20 -05:00
Allison Vacanti
6bb22b952c Add option parsing to NVBENCH_MAIN.
- Convert benchmark_manager into a read-only structure.
  - Mutable benchmarks will be provided by
    `option_parser::get_benchmarks()` or
    `benchmark_manager::clone_benchmarks()`.
2021-02-04 13:15:21 -05:00
Allison Vacanti
efd4442d1b Add option_parser.
Currently supports `--benchmark` and `--axis` options.
2021-02-03 21:39:17 -05:00
Allison Vacanti
b48da531b0 Implement/test mutable string_axis. 2021-02-03 13:37:01 -05:00
Allison Vacanti
e732e50120 Implement/test mutable int64_axis. 2021-02-03 13:32:19 -05:00
Allison Vacanti
36de02d2be Implement/test mutable float64_axis. 2021-02-03 12:50:50 -05:00
Allison Vacanti
6aac1554d5 Add type axis masking.
Allow types in a type_axis to be masked out, skipping all
benchmark configurations that use a masked out type.

This will allow the axis to be modified at runtime with CLI args.

Requires various changes to support mutable axes.
2021-02-02 18:24:26 -05:00
Allison Vacanti
6dd4bebe6e Clonable benchmarks.
Adds clone() methods to benchmark_base and axis_base, along with the
scaffolding to support them.
2021-01-14 17:51:13 -05:00
Allison Vacanti
0f3f1ef899 Polishing up the proof of concept.
- Enable throughput stats
- Add cold measurements
- Print liveness/log messages while running trials.
- Improve markdown formatting
- nvbench::range
2021-01-02 01:45:12 -05:00
Allison Vacanti
93ed09f2b3 Finish off proof of concept level functionality.
Add `measure_hot` and `markdown_format` as detail classes, since
they're still pretty rough and need to be worked into the design.
But for now, they're functional proofs of concept. The formatting is
rough, but demonstrates basics.

Also added launch, exec, and summary functionality. These aren't details
and will be part of the project's API.
2021-01-01 21:47:20 -05:00
Allison Vacanti
3ffd2e6aea Add NVBENCH_CREATE and associated machinery. 2021-01-01 01:36:53 -05:00
Allison Vacanti
0776bdc4be Add nvbench::runner. 2020-12-31 21:27:51 -05:00
Allison Vacanti
29d7eafc33 cartesian_product<type_list<...>> -- now with edge cases!
Add test cases for zero or one input type_list.
2020-12-31 19:51:18 -05:00
Allison Vacanti
b1cbf6cab8 Add test for typeless state_generator::create.
Turns out this edgecase was already handled correctly, but let's
make sure it stays that way.
2020-12-31 17:07:19 -05:00
Allison Vacanti
a1bd7b03b3 Update state_generator::create to handle type axes. 2020-12-31 16:28:41 -05:00
Allison Vacanti
fd3b82cc6f Move sleep kernel to its own header.
This will be useful for testing in general.
2020-12-30 14:47:25 -05:00
Allison Vacanti
ad44463d6e Replace params class with nvbench::named_values.
Refactor nvbench::state to use this for axis parameters.

These will also be useful for summaries and measurements.

Also adds a new ASSERT_THROWS_ANY macro to test some of the new API.
2020-12-30 14:45:46 -05:00
Allison Vacanti
8c0b8e3423 Add cpu_timer. 2020-12-29 23:51:09 -05:00
Allison Vacanti
b07ffafff4 Add cuda_timer, cuda_stream. 2020-12-29 23:50:39 -05:00
Allison Vacanti
981031e732 Update axes_metadata test to reflect new API. 2020-12-29 19:41:29 -05:00
Allison Vacanti
e631f1ff03 Add state_generator::create method to encapsulate state creation. 2020-12-29 19:35:07 -05:00
Allison Vacanti
beaead2c3f Split benchmark into more specialized, nontemplated structs. 2020-12-29 19:34:11 -05:00
Allison Vacanti
093077de5f Add nvbench::state.
This class holds a single value for each runtime axis.
2020-12-27 10:44:22 -05:00
Allison Vacanti
7b14ceb3fe Add detail::state_generator.
This helper utility computes the cartesian product of the runtime
axes.
2020-12-27 10:29:24 -05:00
Allison Vacanti
40f92b4705 Add initial nvbench::benchmark.
It's basically just a container for the various axis classes at this
point.
2020-12-24 17:33:03 -05:00
Allison Vacanti
691ed2c18d Add nvbench::params. 2020-12-22 17:44:33 -05:00