mirror of
https://github.com/NVIDIA/nvbench.git
synced 2026-05-27 08:25:56 +00:00
338936b6fe
Implements `cuda.bench.results.BenchmarkResult` class to represent data from JSON output of benchmark execution.
The contains implements two class methods `BenchmarkResult.from_json(filename : str | os.PathLike, *, metadata : Any = None)` which expects well-formed JSON filename and `BenchmarkResult.empty(*, metadata : Any = None)` intended to represent failed result with reasons that can be recorded in metadata at user's discretion.
The `BenchmarkResult` implements mapping interface, supporting `.keys()`, `.values()`, `.items()` methods, `__len__`, `__contains__`, `__getitem__` and `__iter__` special methods.
Values in `BenchmarkResult` has type `cuda.bench.results.SubBenchmarkResult` which implements a list-like interface, i.e. implements `__len__`, `__getitem__`, and `__iter__` special methods. Values in this list-like structure correspond to measurements of individual states of a particular benchmark (the key in `BenchmarkResult`).
Elements of `SubBenchmarkResult` structure have type `SubBenchmarkState` that supports mapping protocol with axis_values as a key and represent data corresponding to measurements for a particular state (combination of settings for each axis).
The state provides `.samples` and `.frequencies` attributes storing raw execution duration values and estimates for average GPU frequencies.
Example usage:
```
import array, numpy as np, cuda.bench.results
r = cuda.bench.results.BenchmarkResult("perf_data/axes_run1.json")
r["copy_sweep_grid_shape"].centers_with_frequencies(
lambda t, f: np.median(np.asarray(t)*np.asarray(f)))
```
```
In [1]: import array, numpy as np, cuda.bench.results
In [2]: r = cuda.bench.results.BenchmarkResult("temp_data/axes_run1.json")
In [3]: list(r)
Out[3]:
['simple',
'single_float64_axis',
'copy_sweep_grid_shape',
'copy_type_sweep',
'copy_type_conversion_sweep',
'copy_type_and_block_size_sweep']
In [4]: r["simple"].centers(lambda t: np.percentile(t, [25,75]))
Out[4]: {'Device=0': array([0.00100966, 0.00101299])}
In [5]: r.centers(lambda t: np.percentile(t, [25,75]))["simple"]
Out[5]: {'Device=0': array([0.00100966, 0.00101299])}
In [6]: len(r)
Out[6]: 6
In [7]: "fake" in r
Out[7]: False
```
Each `SubBenchmarkState` implements
`.summaries` attribute - rich object that retains tag/name/hint/hide/description metadata.
* Add nvbench-json-summary to render NVBench JSON output as an NVBench-style
markdown summary table, including axis formatting, device sections, hidden
summary filtering, and summary hint formatting.
Update packaging, type stubs, and tests for the new namespace, renamed
classes, Python 3.10-compatible annotations, and summary-table generation.
* Split tests in test_benchmark_result into smaller tests
* Fix break due to file name change
* Add python/examples/benchmark_result_autotune.py
This example demonstrates using cuda.bench and cuda.bench.results
to implement simple auto-tuning, demonstrated on selecting of
tile shape hyperparameter for naive stencil kernel implemented
in numba-cuda.
* Resolve ruff PLE0604
* Fix for format_axis_value in json format script to handle None value
Add tests to cover such input.
* Address code rabbit review feedback
* Fix license header, add validation
* Addressed both issues raised in review
Malformed values are now represented in result as None.
Skipped benchmarks are no longer dropped, i.e., they are present
in BenchmarkResult data, but they are not reflected in summary
table in line with what NVBench-instrumented benchmarks do.
51 lines
1.8 KiB
Bash
Executable File
51 lines
1.8 KiB
Bash
Executable File
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
# Target script for `docker run` command in test_cuda_bench.sh
|
|
# The /workspace pathnames are hard-wired here.
|
|
|
|
# Install GCC 13 toolset (needed for builds that might happen during testing)
|
|
/workspace/ci/util/retry.sh 5 30 dnf -y install gcc-toolset-13-gcc gcc-toolset-13-gcc-c++
|
|
echo -e "#!/bin/bash\nsource /opt/rh/gcc-toolset-13/enable" >/etc/profile.d/enable_devtools.sh
|
|
source /etc/profile.d/enable_devtools.sh
|
|
|
|
# Set up Python environment (only if not already available)
|
|
source /workspace/ci/pyenv_helper.sh
|
|
if ! command -v python${py_version} &> /dev/null; then
|
|
setup_python_env "${py_version}"
|
|
fi
|
|
|
|
# Upgrade pip
|
|
python -m pip install --upgrade pip
|
|
|
|
echo "Python version: $(python --version)"
|
|
echo "CUDA version: $(nvcc --version | grep release)"
|
|
|
|
# Wheel should be in /workspace/wheelhouse (downloaded by workflow or built locally)
|
|
WHEELHOUSE_DIR="/workspace/wheelhouse"
|
|
|
|
# Find the cuda-bench wheel (multi-CUDA wheel)
|
|
# Prefer manylinux wheels, fall back to any wheel
|
|
CUDA_BENCH_WHEEL_PATH="$(ls ${WHEELHOUSE_DIR}/cuda_bench-*manylinux*.whl 2>/dev/null | head -1)"
|
|
if [[ -z "$CUDA_BENCH_WHEEL_PATH" ]]; then
|
|
CUDA_BENCH_WHEEL_PATH="$(ls ${WHEELHOUSE_DIR}/cuda_bench-*.whl 2>/dev/null | head -1)"
|
|
fi
|
|
|
|
if [[ -z "$CUDA_BENCH_WHEEL_PATH" ]]; then
|
|
echo "Error: No cuda-bench wheel found in ${WHEELHOUSE_DIR}"
|
|
echo "Contents of ${WHEELHOUSE_DIR}:"
|
|
ls -la ${WHEELHOUSE_DIR}/ || true
|
|
exit 1
|
|
fi
|
|
|
|
# Determine which CUDA extra to install (defaults to cu12 if not specified)
|
|
CUDA_EXTRA="${cuda_extra:-cu${cuda_version}}"
|
|
TEST_EXTRA="test-cu${cuda_version}"
|
|
|
|
echo "Installing wheel: $CUDA_BENCH_WHEEL_PATH with extras: ${TEST_EXTRA}"
|
|
python -m pip install "${CUDA_BENCH_WHEEL_PATH}[${TEST_EXTRA}]"
|
|
|
|
# Run tests
|
|
cd "/workspace/python/test/"
|
|
python -m pytest -v .
|