mirror of
https://github.com/Comfy-Org/comfy-aimdo.git
synced 2026-11-04 23:37:21 +00:00
[PR #46] [MERGED] Implement threaded loader components (CORE-43) #85
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/comfy-aimdo#85
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
📋 Pull Request Information
Original PR: https://github.com/Comfy-Org/comfy-aimdo/pull/46
Author: @rattus128
Created: 5/6/2026
Status: ✅ Merged
Merged: 5/12/2026
Merged by: @alexisrolland
Base:
master← Head:dev/threaded-load📝 Commits (10+)
947b1a3host_buffer: plan api change to grow the buffer6c8c04csketch prewarm hint54b5eaastub file reader5cc2abasketch truncate5de82a4dispatch: Add cuMemHostRegister/Unregister5ef6cacthreads APIa405f38xfer file0475692hostbuf plat8156b25prewarm3853bd6implement main hostbuf protocol📊 Changes
19 files changed (+881 additions, -17 deletions)
View changed files
📝
comfy_aimdo/host_buffer.py(+62 -6)📝
src-cuda/dispatch.c(+2 -0)📝
src-hip/dispatch.c(+10 -0)➕
src-posix/hostbuf-plat.c(+42 -0)➕
src-posix/thread-plat.c(+71 -0)➕
src-posix/xfer-file-plat.c(+24 -0)➕
src-win/hostbuf-plat.c(+35 -0)➕
src-win/thread-plat.c(+68 -0)➕
src-win/xfer-file-plat.c(+40 -0)📝
src/control.c(+7 -0)📝
src/gpu_dispatch.h(+5 -0)➕
src/hostbuf-plat.h(+12 -0)➕
src/hostbuf-prewarm.c(+98 -0)➕
src/hostbuf-prewarm.h(+6 -0)📝
src/hostbuf.c(+198 -11)📝
src/plat.h(+2 -0)➕
src/thread-plat.h(+33 -0)➕
src/xfer-file.c(+154 -0)➕
src/xfer-file.h(+12 -0)📄 Description
Contribution Agreement
Currently loading models from disk has a bad mix of single-core CPU and disk activity serialization.
So create two thread pools to:
1: Prefault memory in the background while the main thread continues.
2: Do chunked + threaded transfers from disk to RAM.
If you prefault, cudaHostRegister is really fast (way way faster than cudaHostMemAlloc). So we use the background prefault of some headroom to completely hide this latency. The chunked transfer is significantly faster pageable cudaMemcpyAsync.
In addition to the above, also expand the hostbuf API to be growable with the above prefaulter.
So the usage pattern for this is use the new hostbuf APIs to create temp buffers that are pinned and always transfer via said pinned memory.
With accompanying ComfyUI changes I get the following preliminary speedups.
Windows RTX5060, 64GB, PCIE4x4 NVME, LTX2.3 FP8 (no Lora) 360P, first run on launch:
Before:
After:
Linux RTX5090, 96GB, PCIE5x4 NVME, LTX2.3 FP8 (no Lora), 720P, first run on launch, caches cold:
Before:
After:
This is nsight of the model load on a warm load. The gaps between the green blocks are about what you would expect, given the GPU can DMA master than the memcpy to pin.
More data to come Comfy side, along with how to use this in code.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.