kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2843] [TransferEngine] TcpTransport benchmark note: VRAM-backed transfers pay per-chunk staging; original "2x host-path loss on USE_CUDA" was a benchmark-config artifact #737

New issue

Closed

opened 2026-10-29 21:18:53 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:18:53 +00:00

Owner

Originally created by @n-WN on GitHub (Jul 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2843

Bug Report / Performance

TcpTransport throughput drops ~2x when the build enables USE_CUDA, even for pure host-memory transfers — measured on loopback with transfer_engine_bench (write, 4 threads × batch 32, duration 3–4 s, same box, same commit f1f5e40, only the build flag differs):

block size	build without CUDA	build with `-DUSE_CUDA=ON`
64 KB	0.99 GB/s	0.84–0.88 GB/s
256 KB	2.10 GB/s	1.09–1.28 GB/s
1 MB	3.41 GB/s	1.53–1.64 GB/s

All buffers are plain DRAM (numa_alloc/malloc); no GPU memory is involved in the transfer.

Root cause (from reading `tcp_transport.cpp`)

Under USE_CUDA (and the other accelerator macros), both ClientSession::writeBody/readBody and ServerSession::writeBody/readBody call getCudaDeviceId(addr) → cudaPointerGetAttributes() once per 64 KB chunk (not once per request), on both ends of the connection. For a 1 MB transfer that is 16 chunks × 2 sides = 32 CUDA runtime calls per request; cudaPointerGetAttributes costs on the order of a microsecond and serializes on CUDA runtime internals, which is enough to halve throughput at these block sizes. The result of the lookup cannot change between chunks of one request — the address range is fixed at submit time.

Suggested fix

Resolve the memory type once per request (at header processing / initiate) and carry cuda_device through the chunk loop, instead of re-querying per chunk. That is a small, behavior-preserving change; a further step could consult the registered-buffer metadata (the engine already knows the location string of registered memory) and skip the CUDA query entirely for buffers registered as cpu:*.

Happy to pick this up as a follow-up to my in-flight TCP work if nobody else wants it — the benchmark setup above reproduces it in a couple of minutes on any CUDA-build machine.

Environment

main @ f1f5e40; x86_64 Linux with a recent GCC/CUDA toolchain; loopback P2PHANDSHAKE metadata.

Originally created by @n-WN on GitHub (Jul 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2843 ### Bug Report / Performance **TcpTransport throughput drops ~2x when the build enables USE_CUDA, even for pure host-memory transfers** — measured on loopback with `transfer_engine_bench` (write, 4 threads × batch 32, duration 3–4 s, same box, same commit `f1f5e40`, only the build flag differs): | block size | build without CUDA | build with `-DUSE_CUDA=ON` | |---|---|---| | 64 KB | 0.99 GB/s | 0.84–0.88 GB/s | | 256 KB | 2.10 GB/s | 1.09–1.28 GB/s | | 1 MB | **3.41 GB/s** | **1.53–1.64 GB/s** | All buffers are plain DRAM (`numa_alloc`/`malloc`); no GPU memory is involved in the transfer. ### Root cause (from reading `tcp_transport.cpp`) Under `USE_CUDA` (and the other accelerator macros), both `ClientSession::writeBody`/`readBody` and `ServerSession::writeBody`/`readBody` call `getCudaDeviceId(addr)` → `cudaPointerGetAttributes()` **once per 64 KB chunk** (not once per request), on both ends of the connection. For a 1 MB transfer that is 16 chunks × 2 sides = 32 CUDA runtime calls per request; `cudaPointerGetAttributes` costs on the order of a microsecond and serializes on CUDA runtime internals, which is enough to halve throughput at these block sizes. The result of the lookup cannot change between chunks of one request — the address range is fixed at submit time. ### Suggested fix Resolve the memory type **once per request** (at header processing / `initiate`) and carry `cuda_device` through the chunk loop, instead of re-querying per chunk. That is a small, behavior-preserving change; a further step could consult the registered-buffer metadata (the engine already knows the location string of registered memory) and skip the CUDA query entirely for buffers registered as `cpu:*`. Happy to pick this up as a follow-up to my in-flight TCP work if nobody else wants it — the benchmark setup above reproduces it in a couple of minutes on any CUDA-build machine. ### Environment - main @ `f1f5e40`; x86_64 Linux with a recent GCC/CUDA toolchain; loopback P2PHANDSHAKE metadata.

bsakaguchi

2026-10-29 21:18:53 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:18:54 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 10, 2026):

Thanks for opening this issue, @n-WN!

Field	Value
Issue	#2843
GitHub user ID	`30841158`
Reporter	@n-WN

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 10, 2026): Thanks for opening this issue, @n-WN! | Field | Value | |-------|-------| | **Issue** | #2843 | | **GitHub user ID** | `30841158` | | **Reporter** | @n-WN | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:54 +00:00

Author

Owner

@he-yufeng commented on GitHub (Jul 12, 2026):

Confirmed the root cause from a read of tcp_transport.cpp. In the server read path, getCudaDeviceId(addr) is called inside the per-chunk block (around line 261) — and addr is the constant buffer base, only total_transferred_bytes_ advances between chunks. So every 64 KB chunk re-runs cudaPointerGetAttributes on the same pointer, which is exactly the overhead you measured; a CUDA build pays it once per chunk instead of once per transfer.

The fix is to resolve the device id once when the transfer starts and reuse it for every chunk. The same per-chunk getCudaDeviceId(addr) pattern appears at the other transfer sites (~192, ~399, ~503), so they'd want the same treatment.

I verified this by reading, not benchmarking (no GPU on hand), so the throughput numbers are yours — but the redundant call is clearly on the hot path. Happy to review a PR that hoists it.

@he-yufeng commented on GitHub (Jul 12, 2026): Confirmed the root cause from a read of `tcp_transport.cpp`. In the server read path, `getCudaDeviceId(addr)` is called inside the per-chunk block (around line 261) — and `addr` is the constant buffer base, only `total_transferred_bytes_` advances between chunks. So every 64 KB chunk re-runs `cudaPointerGetAttributes` on the same pointer, which is exactly the overhead you measured; a CUDA build pays it once per chunk instead of once per transfer. The fix is to resolve the device id once when the transfer starts and reuse it for every chunk. The same per-chunk `getCudaDeviceId(addr)` pattern appears at the other transfer sites (~192, ~399, ~503), so they'd want the same treatment. I verified this by reading, not benchmarking (no GPU on hand), so the throughput numbers are yours — but the redundant call is clearly on the hot path. Happy to review a PR that hoists it.

bsakaguchi commented

2026-10-29 21:18:55 +00:00

Author

Owner

@n-WN commented on GitHub (Jul 12, 2026):

Correction: I re-ran the attribution today and the original report does not hold up. The ~2x delta was a benchmark-configuration artifact, not a host-path regression. @he-yufeng — the code fact you confirmed (the query really is per-chunk and redundant) is true, but measurements show it cannot be the cause; details below.

1. The per-chunk call is ~0.3%, not ~50%. Microbenchmark on the same H100 box (context initialized, 200k iters):

pointer class	`cudaPointerGetAttributes` per call
plain `malloc` (unregistered host — the class the TCP path queries)	0.125 µs
`cudaMallocHost` / `cudaHostRegister` / `cudaMalloc`	0.07–0.10 µs
unregistered host, different page every call (defeats driver caching)	0.126 µs

At 64 KB per chunk that is ~0.3% of chunk wire time even counting both ends — two orders of magnitude short of explaining 2x.

2. The original A/B compared VRAM buffers against host buffers without knowing it. transfer_engine_bench defines use_vram only under the accelerator macros, with default true — the flag does not even exist in a non-CUDA build (passing it errors out). So with the flag omitted from both command lines, the CUDA build silently benchmarked cudaMalloc VRAM buffers (per-chunk staging: new + synchronous cudaMemcpy + delete on every 64 KB), while the non-CUDA build used numa_alloc host memory. My "all buffers are plain DRAM" claim in the report was wrong for the CUDA column. Same binary, same box today: VRAM 1.32–1.66 GB/s vs --use_vram=false 1.73–2.13 GB/s @ 1 MB.

3. With the configuration fixed, there is no USE_CUDA host-path regression. Equal-flags rebuild (only -DUSE_CUDA flipped, same commit), host memory both sides, interleaved runs on the same box:

block	non-CUDA build	CUDA build, `--use_vram=false`
1 MB	2.44–2.83 GB/s	1.95–2.32 GB/s
256 KB	1.92–2.09 GB/s	1.77–2.16 GB/s

Overlapping ranges, within run-to-run variance (absolute numbers differ from the original report — different day/load on a shared box — but the interleaved comparison is internally controlled).

Consequence: given these numbers, the hoist I offered in the description would be measurable-benefit-free churn, so I won't send one. The honest disposition is to close this as not-a-bug (mislabeled benchmark configuration), or retitle it as a note that VRAM-backed TCP transfers pay a per-chunk staging cost — which is the real (and expected) difference the numbers were showing. Maintainers' call; apologies for the noise, and thanks @he-yufeng for engaging — the reading was right, the magnitude was not.

@n-WN commented on GitHub (Jul 12, 2026): **Correction: I re-ran the attribution today and the original report does not hold up. The ~2x delta was a benchmark-configuration artifact, not a host-path regression.** @he-yufeng — the code fact you confirmed (the query really is per-chunk and redundant) is true, but measurements show it cannot be the cause; details below. **1. The per-chunk call is ~0.3%, not ~50%.** Microbenchmark on the same H100 box (context initialized, 200k iters): | pointer class | `cudaPointerGetAttributes` per call | |---|---| | plain `malloc` (unregistered host — the class the TCP path queries) | 0.125 µs | | `cudaMallocHost` / `cudaHostRegister` / `cudaMalloc` | 0.07–0.10 µs | | unregistered host, different page every call (defeats driver caching) | 0.126 µs | At 64 KB per chunk that is ~0.3% of chunk wire time even counting both ends — two orders of magnitude short of explaining 2x. **2. The original A/B compared VRAM buffers against host buffers without knowing it.** `transfer_engine_bench` defines `use_vram` **only under the accelerator macros, with default `true`** — the flag does not even exist in a non-CUDA build (passing it errors out). So with the flag omitted from both command lines, the CUDA build silently benchmarked **`cudaMalloc` VRAM buffers** (per-chunk staging: `new` + synchronous `cudaMemcpy` + `delete` on every 64 KB), while the non-CUDA build used `numa_alloc` host memory. My "all buffers are plain DRAM" claim in the report was wrong for the CUDA column. Same binary, same box today: VRAM 1.32–1.66 GB/s vs `--use_vram=false` 1.73–2.13 GB/s @ 1 MB. **3. With the configuration fixed, there is no USE_CUDA host-path regression.** Equal-flags rebuild (only `-DUSE_CUDA` flipped, same commit), host memory both sides, interleaved runs on the same box: | block | non-CUDA build | CUDA build, `--use_vram=false` | |---|---|---| | 1 MB | 2.44–2.83 GB/s | 1.95–2.32 GB/s | | 256 KB | 1.92–2.09 GB/s | 1.77–2.16 GB/s | Overlapping ranges, within run-to-run variance (absolute numbers differ from the original report — different day/load on a shared box — but the interleaved comparison is internally controlled). **Consequence:** given these numbers, the hoist I offered in the description would be measurable-benefit-free churn, so I won't send one. The honest disposition is to close this as not-a-bug (mislabeled benchmark configuration), or retitle it as a note that VRAM-backed TCP transfers pay a per-chunk staging cost — which is the real (and expected) difference the numbers were showing. Maintainers' call; apologies for the noise, and thanks @he-yufeng for engaging — the reading was right, the magnitude was not.

bsakaguchi commented

2026-10-29 21:18:55 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 11, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 11, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:18:55 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:17 +00:00

[PR #737] [MERGED] [TransferEngine] exclude packaging ascend precompiled libraries #1780