mirror of
https://github.com/NVIDIA/nvbench.git
synced 2026-07-13 02:27:22 +00:00
d483a5bc73
* Replace forwarding with semantically more accurate std::move Also add comment within percentile_rank to document precondition on input values checked with assert statement. Also, sharpened the comment around percentile_rank function * Duplicate-heavy boundary test is added Prepare duplicate heavy input and check sort-based quartile computation result with selection-based one. std::nth_element only guarantees that the nth element is the value that would appear there in sorted order; it does not fully sort equal partitions. Bugs in the selection implementation, especially when selecting Q1 from the left half and Q3 from the right half after selecting the median, are more likely to show up when many samples equal the quartile values. * Generate cold summaries only if some accepted samples have been accumulated Cold measurement can discard throttled trials before incrementing the accepted sample count, then stop on timeout with zero recorded samples. In that case, only emit the sample-size summary and skip derived timing, bandwidth, clock, and bulk summaries that require accepted samples. This avoids divide-by-zero mean calculations and quartile/IQR computation over empty sample vectors. Keep timeout diagnostics reachable for zero-sample runs and add an explicit warning when no accepted cold samples were recorded. Factor timeout warning emission into a private helper so the zero-sample and normal paths share the same diagnostic logic. Suppress low-sample relative stdev noise Add a statistics helper that returns no relative standard-deviation noise until there are enough samples for a meaningful estimate. Use it for cold CPU/GPU and CPU-only summaries so the low-sample +inf stdev sentinel is not published as real relative noise or used for max-noise timeout warnings. Add statistics coverage for suppressing the low-sample sentinel and computing relative stdev noise once the sample threshold is reached. compute_standard_deviation_noise return nullopt if standard deviation is not finite Test verify that noise is nullopt when not enough samples are accumulated Added statistics::has_enough_samples_for_noise_estimate(...) Used it in standard_deviation, compute_standard_deviation_noise, compute_robust_noise. Added timeout diagnostics in cold and CPU-only paths. if max-noise is configured and the run timed out before enough samples exist to estimate noise, the log now says that explicitly, otherwise the existing “over noise threshold” warning remains unchanged. Added a statistics test assertion for the new sample-count predicate. * Test quartile values across selection threshold Add fixed expected-value assertions for quartile tests around the sort/selection switch point, including duplicate-heavy inputs. This keeps the tests from only proving that both implementations agree with each other. * Add NaN guards to percentiles and quartiles computation routines * Add tests for handling of NaNs in quartile routine inputs * Add comment re magic sort/select threshold value * Refactor logic of emitting warnings between cold and cpu-only measures Introduce new header file with inline implementation. Use it from measure_cold.cuh and measure_cpu_only.cxx * Add static assertion that ValueType is a floating-point type * Check consistency of sort- vs. select-based quartiles using threshold constant Expose quartile threshold value, use it in testing to test around that value. * Collapsed two branches with identical bodies * test_compute_standard_deviation_noise exercises other invalid inputs * Preserve stdev noise summaries for low sample counts Keep legacy stdev/relative summary tags present even when too few samples are available to compute a meaningful standard-deviation noise estimate. Use the standard-deviation unavailable sentinel for those values so existing summary consumers continue to see the expected tags. Factor the sentinel into the statistics helpers and use it from both standard_deviation() and stdev_noise_or_sentinel(), keeping the schema compatibility behavior explicit and tested. * timeout_warnings now treats engaged NaN and negative stdev noise as unavailable Add a focused test target, nvbench.test.measure_timeout_warnings, covering: - NaN stdev noise -> “unable to estimate noise” - negative stdev noise -> “unable to estimate noise” - +inf stdev noise -> “over noise threshold” * Test nullopt explicitly in warning check test check_noise_warning() now takes std::optional<nvbench::float64_t>, matching the production helper, and the test now covers std::nullopt explicitly in addition to NaN, negative, and +inf. * Tighten statistics and timeout warning tests Document that percentile helpers return quiet NaNs for NaN-containing inputs. Make quartile expected-value tests compute ranks from the documented round(p / 100 * (n - 1)) rule instead of reusing statistics::percentile_rank(), so rank regressions are caught independently. Extend timeout-warning coverage to exercise the too-few-samples max-noise path in addition to unavailable, invalid, and infinite stdev-noise inputs. * Test for timeout warnings for min-samples and min-time
119 lines
3.9 KiB
Plaintext
119 lines
3.9 KiB
Plaintext
/*
|
|
* Copyright 2026 NVIDIA Corporation
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 with the LLVM exception
|
|
* (the "License"); you may not use this file except in compliance with
|
|
* the License.
|
|
*
|
|
* You may obtain a copy of the License at
|
|
*
|
|
* http://llvm.org/foundation/relicensing/LICENSE.txt
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
|
|
#include <nvbench/detail/measure_timeout_warnings.cuh>
|
|
#include <nvbench/detail/statistics.cuh>
|
|
#include <nvbench/printer_base.cuh>
|
|
#include <nvbench/stopping_criterion.cuh>
|
|
#include <nvbench/types.cuh>
|
|
|
|
#include <cmath>
|
|
#include <limits>
|
|
#include <optional>
|
|
#include <sstream>
|
|
#include <string>
|
|
#include <utility>
|
|
#include <vector>
|
|
|
|
#include "test_asserts.cuh"
|
|
|
|
struct recording_printer : nvbench::printer_base
|
|
{
|
|
explicit recording_printer(std::ostream &stream)
|
|
: nvbench::printer_base{stream}
|
|
{}
|
|
|
|
std::vector<std::pair<nvbench::log_level, std::string>> logs;
|
|
|
|
protected:
|
|
void do_log(nvbench::log_level level, const std::string &message) override
|
|
{
|
|
logs.emplace_back(level, message);
|
|
}
|
|
};
|
|
|
|
void check_noise_warning(
|
|
std::optional<nvbench::float64_t> stdev_noise,
|
|
const std::string &expected_message,
|
|
nvbench::int64_t total_samples = nvbench::detail::statistics::min_samples_for_noise_estimate)
|
|
{
|
|
std::ostringstream stream;
|
|
recording_printer printer{stream};
|
|
nvbench::criterion_params params;
|
|
params.set_float64("max-noise", 0.01);
|
|
|
|
nvbench::detail::log_measurement_timeout_warnings(printer,
|
|
params,
|
|
1.0,
|
|
total_samples,
|
|
1,
|
|
1.0,
|
|
stdev_noise);
|
|
|
|
ASSERT(printer.logs.size() == 1);
|
|
ASSERT(printer.logs[0].first == nvbench::log_level::warn);
|
|
ASSERT(printer.logs[0].second.find(expected_message) != std::string::npos);
|
|
}
|
|
|
|
void test_non_finite_or_invalid_stdev_noise_timeout_warning()
|
|
{
|
|
check_noise_warning(std::nullopt,
|
|
"before accumulating enough samples to estimate noise",
|
|
nvbench::detail::statistics::min_samples_for_noise_estimate - 1);
|
|
check_noise_warning(std::nullopt, "unable to estimate noise");
|
|
check_noise_warning(std::numeric_limits<nvbench::float64_t>::quiet_NaN(),
|
|
"unable to estimate noise");
|
|
check_noise_warning(-1.0, "unable to estimate noise");
|
|
check_noise_warning(std::numeric_limits<nvbench::float64_t>::infinity(), "over noise threshold");
|
|
}
|
|
|
|
void test_min_samples_timeout_warning()
|
|
{
|
|
std::ostringstream stream;
|
|
recording_printer printer{stream};
|
|
nvbench::criterion_params params;
|
|
|
|
nvbench::detail::log_measurement_timeout_warnings(printer, params, 1.0, 4, 5, 1.0, std::nullopt);
|
|
|
|
ASSERT(printer.logs.size() == 1);
|
|
ASSERT(printer.logs[0].first == nvbench::log_level::warn);
|
|
ASSERT(printer.logs[0].second.find("before accumulating min_samples") != std::string::npos);
|
|
}
|
|
|
|
void test_min_time_timeout_warning()
|
|
{
|
|
std::ostringstream stream;
|
|
recording_printer printer{stream};
|
|
nvbench::criterion_params params;
|
|
params.set_float64("min-time", 2.0);
|
|
|
|
nvbench::detail::log_measurement_timeout_warnings(printer, params, 1.0, 5, 1, 1.5, std::nullopt);
|
|
|
|
ASSERT(printer.logs.size() == 1);
|
|
ASSERT(printer.logs[0].first == nvbench::log_level::warn);
|
|
ASSERT(printer.logs[0].second.find("before accumulating min_time") != std::string::npos);
|
|
}
|
|
|
|
int main()
|
|
{
|
|
test_non_finite_or_invalid_stdev_noise_timeout_warning();
|
|
test_min_samples_timeout_warning();
|
|
test_min_time_timeout_warning();
|
|
return 0;
|
|
}
|