From e53a1a265466c2a49c14eaf4dcd8645fa09777a0 Mon Sep 17 00:00:00 2001 From: Oleksandr Pavlyk <21087696+oleksandr-pavlyk@users.noreply.github.com> Date: Mon, 4 May 2026 16:11:22 -0500 Subject: [PATCH] Use median and IR/relative as cmp_time/ref_time and cmp_noise/ref_noise These measures are less sensitive to outliers --- python/scripts/nvbench_compare.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/python/scripts/nvbench_compare.py b/python/scripts/nvbench_compare.py index 209e0d1..118e212 100644 --- a/python/scripts/nvbench_compare.py +++ b/python/scripts/nvbench_compare.py @@ -385,16 +385,16 @@ def compare_benches( return next(filter(lambda s: s["tag"] == tag, summaries), None) cmp_time_summary = lookup_summary( - cmp_summaries, "nv/cold/time/gpu/mean" + cmp_summaries, "nv/cold/time/gpu/median" ) ref_time_summary = lookup_summary( - ref_summaries, "nv/cold/time/gpu/mean" + ref_summaries, "nv/cold/time/gpu/median" ) cmp_noise_summary = lookup_summary( - cmp_summaries, "nv/cold/time/gpu/stdev/relative" + cmp_summaries, "nv/cold/time/gpu/ir/relative" ) ref_noise_summary = lookup_summary( - ref_summaries, "nv/cold/time/gpu/stdev/relative" + ref_summaries, "nv/cold/time/gpu/ir/relative" ) # TODO: Use other timings, too. Maybe multiple rows, with a