* Replace forwarding with semantically more accurate std::move
Also add comment within percentile_rank to document precondition
on input values checked with assert statement.
Also, sharpened the comment around percentile_rank function
* Duplicate-heavy boundary test is added
Prepare duplicate heavy input and check sort-based
quartile computation result with selection-based one.
std::nth_element only guarantees that the nth element
is the value that would appear there in sorted order;
it does not fully sort equal partitions. Bugs in the
selection implementation, especially when selecting Q1
from the left half and Q3 from the right half after
selecting the median, are more likely to show up when
many samples equal the quartile values.
* Generate cold summaries only if some accepted samples have been accumulated
Cold measurement can discard throttled trials before incrementing the accepted
sample count, then stop on timeout with zero recorded samples. In that case,
only emit the sample-size summary and skip derived timing, bandwidth, clock, and
bulk summaries that require accepted samples.
This avoids divide-by-zero mean calculations and quartile/IQR computation over
empty sample vectors.
Keep timeout diagnostics reachable for zero-sample runs and add an explicit
warning when no accepted cold samples were recorded. Factor timeout warning
emission into a private helper so the zero-sample and normal paths share the
same diagnostic logic.
Suppress low-sample relative stdev noise
Add a statistics helper that returns no relative standard-deviation noise until
there are enough samples for a meaningful estimate. Use it for cold CPU/GPU and
CPU-only summaries so the low-sample +inf stdev sentinel is not published as
real relative noise or used for max-noise timeout warnings.
Add statistics coverage for suppressing the low-sample sentinel and computing
relative stdev noise once the sample threshold is reached.
compute_standard_deviation_noise return nullopt if standard deviation is not finite
Test verify that noise is nullopt when not enough samples are accumulated
Added statistics::has_enough_samples_for_noise_estimate(...)
Used it in standard_deviation, compute_standard_deviation_noise,
compute_robust_noise.
Added timeout diagnostics in cold and CPU-only paths.
if max-noise is configured and the run timed out before enough
samples exist to estimate noise, the log now says that explicitly,
otherwise the existing “over noise threshold” warning remains
unchanged.
Added a statistics test assertion for the new sample-count
predicate.
* Test quartile values across selection threshold
Add fixed expected-value assertions for quartile tests around the
sort/selection switch point, including duplicate-heavy inputs. This keeps the
tests from only proving that both implementations agree with each other.
* Add NaN guards to percentiles and quartiles computation routines
* Add tests for handling of NaNs in quartile routine inputs
* Add comment re magic sort/select threshold value
* Refactor logic of emitting warnings between cold and cpu-only measures
Introduce new header file with inline implementation. Use it
from measure_cold.cuh and measure_cpu_only.cxx
* Add static assertion that ValueType is a floating-point type
* Check consistency of sort- vs. select-based quartiles using threshold constant
Expose quartile threshold value, use it in testing to test around that value.
* Collapsed two branches with identical bodies
* test_compute_standard_deviation_noise exercises other invalid inputs
* Preserve stdev noise summaries for low sample counts
Keep legacy stdev/relative summary tags present even when too few
samples are available to compute a meaningful standard-deviation noise
estimate. Use the standard-deviation unavailable sentinel for those
values so existing summary consumers continue to see the expected tags.
Factor the sentinel into the statistics helpers and use it from both
standard_deviation() and stdev_noise_or_sentinel(), keeping the schema
compatibility behavior explicit and tested.
* timeout_warnings now treats engaged NaN and negative stdev noise as unavailable
Add a focused test target, nvbench.test.measure_timeout_warnings, covering:
- NaN stdev noise -> “unable to estimate noise”
- negative stdev noise -> “unable to estimate noise”
- +inf stdev noise -> “over noise threshold”
* Test nullopt explicitly in warning check test
check_noise_warning() now takes std::optional<nvbench::float64_t>,
matching the production helper, and the test now covers
std::nullopt explicitly in addition to NaN, negative, and +inf.
* Tighten statistics and timeout warning tests
Document that percentile helpers return quiet NaNs for NaN-containing inputs.
Make quartile expected-value tests compute ranks from the documented
round(p / 100 * (n - 1)) rule instead of reusing statistics::percentile_rank(),
so rank regressions are caught independently.
Extend timeout-warning coverage to exercise the too-few-samples max-noise path
in addition to unavailable, invalid, and infinite stdev-noise inputs.
* Test for timeout warnings for min-samples and min-time
* Add statistics utilities to compute quartiles
Quartiles are computed using nearest rank method.
Two implementations are provided:
1. Sort-based:
a. sort array
b. extract values at ranks of interest
2. Selection based:
a. Run nth_element to find median on whole range
b. Run nth_element on left side to find first quartile
c. Run nth_element on right side to find thirst quartile
Public API copies input into temporary vector which is mutated as needed.
Public API uses sort-based implementation for small arrays ( <= 4096 elements),
and selection-based implementation for larger arrays.
Sort-based implementation can support computation of arbitrary percentiles,
which could be useful later if more extreme statistics is needed.
Add tests covering percentile and quartile edge cases, input iterators,
selection-vs-sorting agreement, empty and singleton inputs, and relative
dispersion validation.
* Add quartiles information to summaries
Use the quartile helpers to report robust cold and CPU-only timing summaries:
Q1, median, Q3, interquartile range, and relative interquartile range.
These values stay hidden.
Summary tags are nv/cold/time/gpu/q1, nv/cold/time/gpu/median,
nv/cold/time/gpu/q3, nv/cold/time/gpu/ir/absolute, nv/cold/time/gpu/ir/relative
ir/absolute = q3 - q1, ir/relative = (q3 - q1)/median
Similar tags added for nv/cold/time/cpu and for CPU-only measures.
Validate relative-dispersion calculations before publishing relative noise
summaries so invalid centers or dispersion values do not produce misleading
summary entries.
* Prefer robust summaries in default output
Only flip visibility for nv/cold/cpu/time, nv/cold/gpu/time,
and nv/cpu_only/only:
- hide mean
- hide stdev/relative
- show median
- show ir/relative
* Use is_close where std::abs(act-exp) was used
* Revert "Prefer robust summaries in default output"
This reverts commit 9a0afc361c.
Basically, all robust statistics summaries entries are hidden,
and mean + stdev/relative are back to be default displayed items
* Address PR review feedback
* Reduce stdrel criterion complexity and ensure termination
Replace the stdrel criterion's growing sample history with an online
mean/variance accumulator. This keeps the stopping criterion based on
relative standard deviation, preserves the unbiased standard-deviation
estimate used for convergence, and reduces per-sample update work from
recomputing over the full history to constant time.
Add a bounded invalid-noise path so measurements that persistently produce
non-finite relative noise, such as all-zero timings, can terminate without
waiting for the wall-time timeout. Keep the normal min-time gate for ordinary
stdrel convergence.
Add focused tests for the online accumulator, stdrel sample-count threshold,
sample-standard-deviation behavior, deterministic convergence inputs, and
persistent invalid-noise termination. Update the CLI help for the stdrel
termination behavior.
* change max-noise to for consistency
* Use online_mean_variance on m_noise_tracker in is_finished()
Previously, standard deviation call was made using current
noise level instead of mean noise level. Because of identity
E[ (N - C)^2 ] =
E[ (N - E[N])^2 ] + (E[N] - C)^2 >= E[ (N - E[N])^2 ]
this led to criterion terminating later than it could have because
the estimated expectation is always greater of equal that the
estimate relative to the mean.
Code used current noise level instead of mean to avoid needing to
make two passed through m_noise_tracker container.
Use of online_mean_variance allows to improve accuracy of estimating
dispersion of noise signal while maintaining single pass through
container.
* Address review feedback
Fixed misleading commit. Introduce private methods to refactor
computation of repeated expressions.
Renamed m_cuda_times_summary to m_measurements_summary, since
criterion can be applied for CPU-only measurements too.
Introduced is_close utility for checking whether two floating
point numbers are closed to one another.
Introduced descriptive constexpr variables for hard-wired
constants
