Tighten missing-dependency install hints so compare-only dependencies point
to cuda-bench[compare] and plotting/dataframe dependencies point to
cuda-bench[plot], instead of defaulting every script to the broader tools
extra.
Also harden nvbench_compare_legacy by reporting missing or skipped state
summaries as UNKNOWN rows instead of silently dropping them, and by converting
missing axis metadata into the existing JSON-structure error path rather than
leaking StopIteration.
Finally, consolidate duplicate finite-number predicates in both compare
scripts so duration formatting and numeric validation share the same helper.
Thread the originating JSON path through bulk timing extraction so sidecar
filenames can be resolved with knowledge of the JSON basename. This lets
nvbench-compare handle --jsonbin output where the filename field may
store the benchmark-process-relative path, such as data/result.json-bin/0.bin,
instead of a path relative to the JSON file directory.
Keep resolution anchored to the JSON location and avoid falling back to the
caller working directory. Add regression coverage for this sidecar layout.
This change helps nvbench_compare properly resolve location of sidecar
binary files in the following CCCL-based workflow:
```
./bin/cub.bench.bitonic_sort.warp_keys.base -d 1 --cold-warmup-runs 16 --min-samples 100 --stopping-criterion entropy --min-r2 0.88 --jsonbin ../../../perf_data/warp-bitonic-sort-run1.json
./bin/cub.bench.bitonic_sort.warp_keys.base -d 1 --cold-warmup-runs 16 --min-samples 100 --stopping-criterion entropy --min-r2 0.88 --jsonbin ../../../perf_data/warp-bitonic-sort-run2.json
PYTHONPATH=./_deps/nvbench-src/python/scripts python ./_deps/nvbench-src/python/scripts/nvbench_compare.py ../../../perf_data/warp-bitonic-sort-run1.json ../../../perf_data/warp-bitonic-sort-run2.json
```
Before this change, script generated numerous RuntimeWarnigns alerting
that absolute path can not be resolved and bulk data will be treated as unavailable
Factor finite-value checks through a common helper so positive and
non-negative finite predicates share the same None/finite guard. Add a
comment explaining the positive lower-bound clamp used for mean/stdev
intervals, since later ratio and log-distance checks require positive
bounds.
Also quote the --axis pow2 example in CLI help so shell users can copy
the example safely.
require_tooling_dependency() now only translates
ModuleNotFoundError when the missing module is the
requested top-level package. Other import
failures are re-raised unchanged.
This helps in situation where third-party dependency
is installed but broken for whatever reason. Previously
we would intercept it and suggest to run
pip install cuda-bench[tools], but that was already done.
Add a shared nvbench_tooling_deps helper that imports packages required
by console tools and reports missing packages with an actionable install
recipe.
Update NVBench scripts to support both direct source-tree execution and
installed package execution through the __package__ import pattern. Defer
third-party tooling imports until they are needed, including lazy loading
for compare tables, device diffs, and plotting paths.
Make loaders resilient to partial initialization failures so a later
retry can complete any dependency that failed previously.
Update tests to cover direct internal use, packaged cuda.bench.scripts
imports, dependency-loader error messages, and cheap CLI validation before
tooling dependencies are loaded.
test_nvbench_tooling_deps.py now has a smoke test that builds a
temporary package layout matching the wheel mapping:
cuda/bench/scripts/nvbench_tooling_deps.py
and imports:
cuda.bench.scripts.nvbench_tooling_deps
That covers the cuda.bench.scripts.* path without requiring a wheel
build/install inside this unit test.
Add a shared nvbench_tooling_deps helper for importing packages required
by NVBench console tools. Missing tooling packages now raise a dedicated
error with an install recipe instead of failing with a raw ImportError.
Update script imports to work both as installed package modules and as
direct source-tree scripts by using the __package__ import pattern for
nvbench_json and the new tooling helper.
Defer nvbench-compare dependencies to the points where they are needed:
NumPy/colorama during normal comparison setup, tabulate during table
rendering, jsondiff only for device mismatch reporting, and plotting
packages only for plot modes.
Update tests to initialize compare tooling when calling internals
directly and add coverage for the tooling dependency loader.
Closes#384
Derive absolute standard deviation from relative stdev and mean when
nv/cold/time/gpu/stdev/absolute is absent. This lets older JSON files
that only contain mean and relative stdev still construct timing
intervals.
Also allow mean/stdev intervals to be built without min/max summaries,
using min/max only as optional clipping bounds when present. This
restores SAME classification for legacy fixture data instead of treating
those rows as missing-interval AMBG cases.
Update nvbench_compare tests to cover derived stdev handling and the
legacy mean/stdev comparison path.
Move Float32BinarySource material-payload detection into the source object.
Default file-backed sources still use resolved file size so missing or empty
sidecars remain unavailable, but positive-count sources with custom readers are
treated as material and proceed through the lazy read path.
Add regression coverage for virtual bulk sources whose custom reader provides
data without a local sidecar file.
Reject boolean summary float payloads instead of coercing them to 1.0/0.0,
while keeping numeric strings accepted for NVBench JSON compatibility.
Add regression coverage for generated bulk-debug Python filenames that require
escaping, and strengthen the plot-along test to assert log-log axes and
confidence-band rendering.
When any --benchmark filter is present, keep comparison limited to the
explicitly selected benchmarks. Leading --axis filters are still replayed onto
each selected benchmark, matching native NVBench option parsing, but they no
longer cause unrelated benchmarks to be compared.
E.g., `-a A=2 -b bench1` now compares only bench1,
`-a A=2 -b bench1 -b bench2` applies A=2 to both selected benchmarks
Update tests for global axis filters with benchmark scopes and document the
selection behavior.
Reject boolean and floating-point values for int64 bulk binary sizes instead of
silently converting them with int(). Keep integer strings accepted for existing
NVBench JSON compatibility, and add regression coverage for valid and malformed
size payloads.
Select robust timing inputs independently for reference and compare data:
prefer robust summaries when present, otherwise recompute robust statistics from
that side's bulk samples. Fall back to mean/stdev summaries only when both sides
cannot provide robust timing inputs.
This allows modern JSON data with robust summaries to compare against legacy JSON
data that lacks robust summaries but includes bulk sample data, without mixing
summary families or unnecessarily falling back to mean/stdev.
Introduce has_robust_interval predicate and used it.
It is used to check whether robust interval can be constructed
before attempting that and falling back to mean/stdev construction
otherwise
1. Skip test when running with Py3.10 if tomli is unavailable
before nvbench_compare fixture is constructed, that is at
collection time rather than at execution time
2. Check that test of plotting options calls plot
3. Test actual output to verify that warning about device
mismatch is absent when device selection is requested
Keep leading --axis filters global even when later --benchmark scopes are
present, so commands like "-a A=2 -b bench" still compare other benchmarks
matching the global axis filter.
Tighten --plot-along validation for the log-log plot path by rejecting
non-numeric, non-positive, and non-finite axis values with targeted errors.
Add regression coverage for global axis scoping and invalid plot-axis values.
Add explicit validation for plot-axis values so string/type axes fail with a
clear CLI error instead of a raw float conversion exception. Add regression
coverage for a type axis.
Route NVBench JSON read failures and missing required root keys through the
documented user-facing error path so malformed inputs return 1 instead of
producing a traceback.
Allow deterministic mean-based timing summaries with zero standard deviation to
form zero-width intervals, while still rejecting negative or non-finite
dispersion values. Reuse the same non-negative finite predicate for relative
noise validation.
Add regression coverage for unreadable inputs, missing root keys, and identical
stable timing summaries.
Let argparse derive the program name from the actual invocation instead of
hardcoding nvbench_compare, so help and error output match the installed
nvbench-compare entry point.
Declare comparison inputs as explicit positional arguments and use parse_args()
instead of parse_known_args(). This preserves --dump-config without input files
while rejecting unknown options through argparse rather than treating typoed
flags as JSON paths.
Add regression coverage for rejecting an unknown CLI option.
Skip UNKNOWN rows when collecting summary plot entries so non-numeric
fractional differences cannot reach the plotting path. Add a regression test
that exercises compare_benches(..., plot=True) with an UNKNOWN row.
Document the supported pow2 axis-filter syntax and update the CLI help example
to use NAME[pow2]=EXP, matching the parser behavior for axes displayed as 2^N.
* Document when status ???? (UNKNOWN) is emitted
* Clarify --no-color behavior
* nvbench_compare.md: clarify --no-color behavior, fix example
* Document display options in nvbench_compare.md
* Small mention of plotting capabilities in nvbench_compare.md
* Call out that example requires shell with process substitution capabilities
Test file changed to use get_default_thresholds() function instead
of call to constructor.
This is to make sure that default preset values do not diverge from
values encoded in the constructor.
* Define nan and inf in generated --bulk-debug-python scripts so pprint output
for non-finite timing values remains valid Python code. Add a regression test
that executes the generated script and verifies nan/inf values round-trip.
* Sharpen bulk-cycle confirmation gating. Only suppress summary-clock
fallback when both reference and compare inputs provide paired, non-empty bulk
sample/frequency payloads. Missing or empty bulk files are treated as
unavailable evidence and still allow sm_clock_rate/mean fallback, while
malformed non-empty payloads continue to produce AMBG.
Add regression coverage for missing bulk files falling back to summary-cycle
confirmation.
These changes resolve automated review feedback
* Register the dynamically loaded nvbench_compare module in sys.modules before
executing it so tests better match normal import behavior.
* Add shared tabulate-capture helpers and select rendered comparison tables by
header suffix instead of relying on the last tabulate call. This makes display
tests robust to future summary or legend table output.
* Add coverage for unusable bulk cycle data forcing an ambiguous result instead
of falling back to summary clock confirmation.
* Rename the TOML parser integration test to clarify that it exercises whichever
parser is available in the environment, and document the Python 3.10 tomli
skip behavior.
When bulk sample or frequency sources are present, do not silently fall
back to summary SM clock confirmation if the bulk cycle data cannot be
used. Report the clear-gap decision as AMBG with a
bulk_cycle_data_unusable reason instead.
Still allow summary-clock fallback when no bulk sample/frequency sources
are present.
Also update the Unknown summary label to describe the broader set of
input-data failures now counted as UNKNOWN.
Treat matched states with unusable timing data as UNKNOWN instead of
dropping them from the comparison. This includes missing, non-finite, or
non-positive timing centers, skipped states, and states with missing GPU
timing summaries.
Add explicit reason codes for these cases so the summary points users at
the underlying data issue. Preserve available timing data from the other
side when only one side is missing, and render unavailable durations as
n/a in all display modes.
Also sort values returned by np.unique_counts before nearest-neighbor
coverage checks so the distance algorithm receives ordered inputs.
Add regression coverage for UNKNOWN counting, skipped states, missing
summaries, unavailable center formatting, and the updated coverage helper.
This permits extracting Python script using Unix CLI tools
when `--bulk-debug-python stdout` is used.
Added example of using this to nvbench_compare.md doc.
Add compact reason labels for explain-mode tables while keeping canonical
reason codes in the undecided summary. Emit a one-line legend only for
non-trivial abbreviations.
Refine interval displays so timing values align across table rows:
- align Lo/Ce/Hi values in explain mode
- align center values in intervals mode when some rows lack interval bounds
- avoid repeating units when center and interval deltas use the same unit
Add a Change column for non-legacy displays so FAST/SLOW rows show the
signed interval-bound relative change, while SAME and UNDECIDED rows remain
blank.
Extend nvbench_compare tests to cover reason legend filtering, interval
alignment, missing-interval alignment, and Change column formatting.
