Apply the same small stability fixes to the modern and legacy compare scripts:
report malformed nested JSON through the structured parse-error path, reject
directory comparisons that produce no matching JSON files, use the supported
pipe table format for old tabulate versions, and retitle the summary plot as a
GPU timing change.
Add regression coverage for the modern comparator error paths and output
formatting.
Emit absolute sidecar paths in bulk-debug rows so generated debug scripts can
be replayed from a different working directory.
Also include duplicate state occurrence information in --plot-along series keys
for both modern and legacy comparators, preventing later duplicate states from
overwriting earlier plotted series.
Add regression coverage for absolute bulk-debug paths and duplicate plot-along
series.
Permit nvbench-compare to use bulk timing samples to confirm SAME even when
sample frequency sidecars are unavailable. This lets legacy --jsonbin outputs,
which recorded sample times but not frequencies, avoid AMBG results when
summary noise is high but timing coverage clearly supports equality.
Cycle coverage is still required when frequency samples are present, and
FAST/SLOW clear-gap confirmation remains cycle-based.
Add regression coverage for noisy bulk samples that exceed the SAME noise
ceiling but are classified SAME from bulk timing/cycle coverage, including the
timing-only legacy-data path.
Tighten missing-dependency install hints so compare-only dependencies point
to cuda-bench[compare] and plotting/dataframe dependencies point to
cuda-bench[plot], instead of defaulting every script to the broader tools
extra.
Also harden nvbench_compare_legacy by reporting missing or skipped state
summaries as UNKNOWN rows instead of silently dropping them, and by converting
missing axis metadata into the existing JSON-structure error path rather than
leaking StopIteration.
Finally, consolidate duplicate finite-number predicates in both compare
scripts so duration formatting and numeric validation share the same helper.
Thread the originating JSON path through bulk timing extraction so sidecar
filenames can be resolved with knowledge of the JSON basename. This lets
nvbench-compare handle --jsonbin output where the filename field may
store the benchmark-process-relative path, such as data/result.json-bin/0.bin,
instead of a path relative to the JSON file directory.
Keep resolution anchored to the JSON location and avoid falling back to the
caller working directory. Add regression coverage for this sidecar layout.
This change helps nvbench_compare properly resolve location of sidecar
binary files in the following CCCL-based workflow:
```
./bin/cub.bench.bitonic_sort.warp_keys.base -d 1 --cold-warmup-runs 16 --min-samples 100 --stopping-criterion entropy --min-r2 0.88 --jsonbin ../../../perf_data/warp-bitonic-sort-run1.json
./bin/cub.bench.bitonic_sort.warp_keys.base -d 1 --cold-warmup-runs 16 --min-samples 100 --stopping-criterion entropy --min-r2 0.88 --jsonbin ../../../perf_data/warp-bitonic-sort-run2.json
PYTHONPATH=./_deps/nvbench-src/python/scripts python ./_deps/nvbench-src/python/scripts/nvbench_compare.py ../../../perf_data/warp-bitonic-sort-run1.json ../../../perf_data/warp-bitonic-sort-run2.json
```
Before this change, script generated numerous RuntimeWarnigns alerting
that absolute path can not be resolved and bulk data will be treated as unavailable
Factor finite-value checks through a common helper so positive and
non-negative finite predicates share the same None/finite guard. Add a
comment explaining the positive lower-bound clamp used for mean/stdev
intervals, since later ratio and log-distance checks require positive
bounds.
Also quote the --axis pow2 example in CLI help so shell users can copy
the example safely.
require_tooling_dependency() now only translates
ModuleNotFoundError when the missing module is the
requested top-level package. Other import
failures are re-raised unchanged.
This helps in situation where third-party dependency
is installed but broken for whatever reason. Previously
we would intercept it and suggest to run
pip install cuda-bench[tools], but that was already done.
Add a shared nvbench_tooling_deps helper that imports packages required
by console tools and reports missing packages with an actionable install
recipe.
Update NVBench scripts to support both direct source-tree execution and
installed package execution through the __package__ import pattern. Defer
third-party tooling imports until they are needed, including lazy loading
for compare tables, device diffs, and plotting paths.
Make loaders resilient to partial initialization failures so a later
retry can complete any dependency that failed previously.
Update tests to cover direct internal use, packaged cuda.bench.scripts
imports, dependency-loader error messages, and cheap CLI validation before
tooling dependencies are loaded.
test_nvbench_tooling_deps.py now has a smoke test that builds a
temporary package layout matching the wheel mapping:
cuda/bench/scripts/nvbench_tooling_deps.py
and imports:
cuda.bench.scripts.nvbench_tooling_deps
That covers the cuda.bench.scripts.* path without requiring a wheel
build/install inside this unit test.
Add a shared nvbench_tooling_deps helper for importing packages required
by NVBench console tools. Missing tooling packages now raise a dedicated
error with an install recipe instead of failing with a raw ImportError.
Update script imports to work both as installed package modules and as
direct source-tree scripts by using the __package__ import pattern for
nvbench_json and the new tooling helper.
Defer nvbench-compare dependencies to the points where they are needed:
NumPy/colorama during normal comparison setup, tabulate during table
rendering, jsondiff only for device mismatch reporting, and plotting
packages only for plot modes.
Update tests to initialize compare tooling when calling internals
directly and add coverage for the tooling dependency loader.
Closes#384
Derive absolute standard deviation from relative stdev and mean when
nv/cold/time/gpu/stdev/absolute is absent. This lets older JSON files
that only contain mean and relative stdev still construct timing
intervals.
Also allow mean/stdev intervals to be built without min/max summaries,
using min/max only as optional clipping bounds when present. This
restores SAME classification for legacy fixture data instead of treating
those rows as missing-interval AMBG cases.
Update nvbench_compare tests to cover derived stdev handling and the
legacy mean/stdev comparison path.
Move Float32BinarySource material-payload detection into the source object.
Default file-backed sources still use resolved file size so missing or empty
sidecars remain unavailable, but positive-count sources with custom readers are
treated as material and proceed through the lazy read path.
Add regression coverage for virtual bulk sources whose custom reader provides
data without a local sidecar file.
Reject boolean summary float payloads instead of coercing them to 1.0/0.0,
while keeping numeric strings accepted for NVBench JSON compatibility.
Add regression coverage for generated bulk-debug Python filenames that require
escaping, and strengthen the plot-along test to assert log-log axes and
confidence-band rendering.
When any --benchmark filter is present, keep comparison limited to the
explicitly selected benchmarks. Leading --axis filters are still replayed onto
each selected benchmark, matching native NVBench option parsing, but they no
longer cause unrelated benchmarks to be compared.
E.g., `-a A=2 -b bench1` now compares only bench1,
`-a A=2 -b bench1 -b bench2` applies A=2 to both selected benchmarks
Update tests for global axis filters with benchmark scopes and document the
selection behavior.
Reject boolean and floating-point values for int64 bulk binary sizes instead of
silently converting them with int(). Keep integer strings accepted for existing
NVBench JSON compatibility, and add regression coverage for valid and malformed
size payloads.
Select robust timing inputs independently for reference and compare data:
prefer robust summaries when present, otherwise recompute robust statistics from
that side's bulk samples. Fall back to mean/stdev summaries only when both sides
cannot provide robust timing inputs.
This allows modern JSON data with robust summaries to compare against legacy JSON
data that lacks robust summaries but includes bulk sample data, without mixing
summary families or unnecessarily falling back to mean/stdev.
Introduce has_robust_interval predicate and used it.
It is used to check whether robust interval can be constructed
before attempting that and falling back to mean/stdev construction
otherwise
1. Skip test when running with Py3.10 if tomli is unavailable
before nvbench_compare fixture is constructed, that is at
collection time rather than at execution time
2. Check that test of plotting options calls plot
3. Test actual output to verify that warning about device
mismatch is absent when device selection is requested
Keep leading --axis filters global even when later --benchmark scopes are
present, so commands like "-a A=2 -b bench" still compare other benchmarks
matching the global axis filter.
Tighten --plot-along validation for the log-log plot path by rejecting
non-numeric, non-positive, and non-finite axis values with targeted errors.
Add regression coverage for global axis scoping and invalid plot-axis values.
Add explicit validation for plot-axis values so string/type axes fail with a
clear CLI error instead of a raw float conversion exception. Add regression
coverage for a type axis.
Route NVBench JSON read failures and missing required root keys through the
documented user-facing error path so malformed inputs return 1 instead of
producing a traceback.
Allow deterministic mean-based timing summaries with zero standard deviation to
form zero-width intervals, while still rejecting negative or non-finite
dispersion values. Reuse the same non-negative finite predicate for relative
noise validation.
Add regression coverage for unreadable inputs, missing root keys, and identical
stable timing summaries.
Let argparse derive the program name from the actual invocation instead of
hardcoding nvbench_compare, so help and error output match the installed
nvbench-compare entry point.
Declare comparison inputs as explicit positional arguments and use parse_args()
instead of parse_known_args(). This preserves --dump-config without input files
while rejecting unknown options through argparse rather than treating typoed
flags as JSON paths.
Add regression coverage for rejecting an unknown CLI option.
Skip UNKNOWN rows when collecting summary plot entries so non-numeric
fractional differences cannot reach the plotting path. Add a regression test
that exercises compare_benches(..., plot=True) with an UNKNOWN row.
Document the supported pow2 axis-filter syntax and update the CLI help example
to use NAME[pow2]=EXP, matching the parser behavior for axes displayed as 2^N.
* Document when status ???? (UNKNOWN) is emitted
* Clarify --no-color behavior
* nvbench_compare.md: clarify --no-color behavior, fix example
* Document display options in nvbench_compare.md
* Small mention of plotting capabilities in nvbench_compare.md
* Call out that example requires shell with process substitution capabilities
Test file changed to use get_default_thresholds() function instead
of call to constructor.
This is to make sure that default preset values do not diverge from
values encoded in the constructor.
