Comfy-Org/comfyui-benchmark

Fork 0

mirror of https://github.com/Comfy-Org/comfyui-benchmark.git synced 2026-11-04 23:47:33 +00:00

[PR #5] feat: capture full benchmark metric set (times, RAM, VRAM, CPU, GPU) on all backends #5

New issue

Open

opened 2026-10-26 22:11:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:11:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyui-benchmark/pull/5
Author: @deepme987
Created: 9/29/2026
Status: 🔄 Open

Base: main ← Head: deepme987/mps-memory-capture

📝 Commits (2)

ea3723b feat: capture device memory on Apple Silicon (MPS) via torch
6f4d83d feat: unified always-on sampler — CPU%, RAM, VRAM, GPU util/power/temp/clocks

📊 Changes

2 files changed (+132 additions, -28 deletions)

View changed files

📝 README.md (+14 -2)
📝 __init__.py (+118 -26)

📄 Description

Summary

Make the node capture everything a benchmark needs, on every backend — times, RAM, VRAM, CPU, and GPU telemetry — so it can serve as the shared metric-collection engine across Desktop / cloud / CLI (a public GPU×workflow benchmark corpus).

Before this, the sampler was nvidia-smi-only: Apple Silicon captured no GPU memory, CPU-only machines got no time series at all, and CPU utilization was never recorded.

What changed (`init.py`)

One always-on system sampler (replaces the NVIDIA-only and MPS-only threads) with a pluggable GPU probe:

Signal	Source	Backends
RAM used/total + CPU utilization %	psutil (`psutil_data`)	all, incl. CPU-only
VRAM, GPU + mem util, temperature, power (draw/instant/limit), SM + memory clocks, PCIe	nvidia-smi (`nvidia_smi_data`)	NVIDIA
Device memory (`current_allocated` / `driver_allocated` bytes)	`torch.mps` (`device_memory_data`)	Apple Silicon
Operation timings (load, CLIP, VAE, sampler total + per-step it/s, total workflow)	ComfyUI hooks	all

Static record — device_info now includes backend (nvidia/mps/cpu), cpu_model, physical/logical core counts, architecture, platform, PyTorch/Python/ComfyUI versions, total VRAM/RAM — so each run is a self-contained corpus record.

device_info.backend ensures unified memory (MPS/CPU, drawn from system RAM) is never ranked against dedicated VRAM.

Config is unchanged and backward-compatible (nvidia-smi.enable_thread / check_interval now gate the whole sampler; the key name is retained).

Testing

python -m py_compile __init__.py nodes.py passes.
Primitives verified on Apple M4 Pro: sysctl machdep.cpu.brand_string → "Apple M4 Pro"; torch.mps.current_allocated_memory / driver_allocated_memory are stable since torch 2.1 (this machine runs torch 2.5.1 + mps).
Manual run gate (needs a live ComfyUI): install into custom_nodes/, run any workflow, and confirm the JSON in outputs/benchmark/ has: populated psutil_data with a 4th cpu_percent column; device_memory_data + device_info.backend: "mps" on Apple Silicon (or nvidia_smi_data with temperature/clocks on NVIDIA); and device_info.cpu_model / core counts / versions.
NVIDIA output is a superset of before (same fields + temperature + clocks); the nvidia path logic is otherwise unchanged.

Timeline visualization of the new series in visualize_benchmark.py is a deliberate follow-up — this PR is capture-only.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyui-benchmark/pull/5 **Author:** [@deepme987](https://github.com/deepme987) **Created:** 9/29/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `deepme987/mps-memory-capture` --- ### 📝 Commits (2) - [`ea3723b`](https://github.com/Comfy-Org/comfyui-benchmark/commit/ea3723bbb71861855952dcf27b81ecfb3950c35f) feat: capture device memory on Apple Silicon (MPS) via torch - [`6f4d83d`](https://github.com/Comfy-Org/comfyui-benchmark/commit/6f4d83d6d81cc73f90d85fac0b97284c8073f021) feat: unified always-on sampler — CPU%, RAM, VRAM, GPU util/power/temp/clocks ### 📊 Changes **2 files changed** (+132 additions, -28 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+14 -2) 📝 `__init__.py` (+118 -26) </details> ### 📄 Description ## Summary Make the node capture **everything a benchmark needs, on every backend** — times, RAM, VRAM, CPU, and GPU telemetry — so it can serve as the shared metric-collection engine across Desktop / cloud / CLI (a public GPU×workflow benchmark corpus). Before this, the sampler was `nvidia-smi`-only: Apple Silicon captured no GPU memory, CPU-only machines got no time series at all, and CPU utilization was never recorded. ## What changed (`__init__.py`) **One always-on system sampler** (replaces the NVIDIA-only and MPS-only threads) with a pluggable GPU probe: | Signal | Source | Backends | |---|---|---| | RAM used/total + **CPU utilization %** | psutil (`psutil_data`) | all, incl. CPU-only | | VRAM, GPU + mem util, **temperature**, power (draw/instant/limit), **SM + memory clocks**, PCIe | nvidia-smi (`nvidia_smi_data`) | NVIDIA | | Device memory (`current_allocated` / `driver_allocated` bytes) | `torch.mps` (`device_memory_data`) | Apple Silicon | | Operation timings (load, CLIP, VAE, sampler total + per-step it/s, total workflow) | ComfyUI hooks | all | **Static record** — `device_info` now includes `backend` (nvidia/mps/cpu), `cpu_model`, physical/logical core counts, architecture, platform, PyTorch/Python/ComfyUI versions, total VRAM/RAM — so each run is a self-contained corpus record. `device_info.backend` ensures unified memory (MPS/CPU, drawn from system RAM) is never ranked against dedicated VRAM. Config is unchanged and backward-compatible (`nvidia-smi.enable_thread` / `check_interval` now gate the whole sampler; the key name is retained). ## Testing - `python -m py_compile __init__.py nodes.py` passes. - Primitives verified on Apple M4 Pro: `sysctl machdep.cpu.brand_string` → "Apple M4 Pro"; `torch.mps.current_allocated_memory` / `driver_allocated_memory` are stable since torch 2.1 (this machine runs torch 2.5.1 + mps). - **Manual run gate** (needs a live ComfyUI): install into `custom_nodes/`, run any workflow, and confirm the JSON in `outputs/benchmark/` has: populated `psutil_data` with a 4th `cpu_percent` column; `device_memory_data` + `device_info.backend: "mps"` on Apple Silicon (or `nvidia_smi_data` with temperature/clocks on NVIDIA); and `device_info.cpu_model` / core counts / versions. - NVIDIA output is a superset of before (same fields + temperature + clocks); the nvidia path logic is otherwise unchanged. Timeline visualization of the new series in `visualize_benchmark.py` is a deliberate follow-up — this PR is capture-only. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:11:23 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfyui-benchmark#5

No description provided.

Rows
Columns