NVIDIA/nvbench
1
0
Fork 0
mirror of https://github.com/NVIDIA/nvbench.git synced 2026-04-20 06:48:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add comment regarding better metrics of failure.

Browse Source
This commit is contained in:
Vyas Ramasubramani
2021-05-26 10:29:53 -07:00
committed by Allison Vacanti
parent 52775574a3
commit ab39c924b2
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
scripts/nvbench_compare.py
View File

@@ -71,6 +71,11 @@ def get_row(cmp_benches, ref_benches):
ref_noise = ref_noise_summary["value"]["value"]
# pass/fail status
# TODO: Currently we're using a very rough metric to determine
# failure by simply adding the standard deviations of the reference
# and sample distributions. Ideally we would use something like
# KL divergence to capture the differences, but that's out of scope
# at this stage.
failed = (cmp_noise - ref_noise) > 2 * (((cmp_noise / 100.) * cmp_time) + ((ref_noise / 100.) * ref_time))
status = (Fore.RED + "FAIL" if failed else Fore.GREEN + "PASS") + Fore.RESET