* Implement warmup-runs count, supported as CLI
CLI option --warmup-runs implemented and documented.
The warm-up counts is enforced to always be positive.
This is necessary to ensure that JIT-ting has occurred,
and use of blocking kernel would not result in time-outs.
Test is option parser is added.
* Ensure that measure_cold::run_warmup instantiates blocking kernel
Because warm-up runs are executed without use of blocking kernel,
the blocking kernel was not jitted until actual measurements were
collected. The module loading cost incurred during the first run
shows as elevated CPU time noise value for the first measurement
as noted in https://github.com/NVIDIA/nvbench/pull/339
This PR adds `this->block_stream(); this->unblock_stream();` prior
to executing warm-up loop with use of blocking kernel disabled.
This ensures that blocking kernel is instantiated during the warm-up,
but it no other kernel is launched between its launch and stream sync
thus avoiding deadlocking.
* Rename --warmup-runs to --cold-warmup-runs, add --cold-max-warmup-walltime
Since configurable number of warmups only applies to measure_cold.cuh
rename the CLI option to reflect that.
Also add --cold-max-warmup-walltime (defaults to -1, i.e. disabled).
If enabled, exits warmup loop before request count is reached if
the wall-time expanded executign warmups exceeds this max-warmup-walltime
value.
These are now owned by the stdrel stopping criterion, and should not be exposed directly in the benchmark/state/etc APIs.
This will affect users that are calling
`NVBENCH_BENCH(...).set_min_time(...)` or
`NVBENCH_BENCH(...).set_max_noise(...)`.
These can be updated to
`NVBENCH_BENCH(...).set_criterion_param_float64(["min-time"|"max-noise"], ...)`.
Human-readable outputs (md) and CLI inputs still use percentages.
In-memory and machine-readable outputs (csv, json) use ratios.
This is the convention that spreadsheet apps expect. Fixes#2.
s/NVBENCH_CREATE/NVBENCH_BENCH/g
s/NVBENCH_BENCH_TEMPLATE/NVBENCH_BENCH_TYPES/g
This will fit nicer once the exec_tags version are added:
NVBENCH_BENCH
NVBENCH_BENCH_TYPES
NVBENCH_BENCH_FLAGS
NVBENCH_BENCH_TYPES_FLAGS
- Add support for single values ("Axis=Value").
- Make other value specs shell friendly:
- Range: "Axis:(2:10:2)" -> "Axis=[2:10:2]"
- List: "Axis:{2,3,4,5}" -> "Axis=[2,3,4,5]"
- ":" -> "=" feels more natural
- "{}()" characters have special meaning in bash.
- "[]" character don't require escapes.
- Using the same braces for both ranges/list is easier to remember,
only the delimiter changes.
- Convert benchmark_manager into a read-only structure.
- Mutable benchmarks will be provided by
`option_parser::get_benchmarks()` or
`benchmark_manager::clone_benchmarks()`.
