[GH-ISSUE #5041] [RFC]: [TE] HipTransport: fast, correct intra-node XGMI for P/D KV transfer on AMD #1276

New issue

Open

opened 2026-10-29 21:24:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:44 +00:00

Owner

Originally created by @Jasen2201 on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5041

Changes proposed

Summary

This RFC tracks a set of changes to classic Transfer Engine's HipTransport (intra-node XGMI on AMD Instinct). The goal is for same-node P/D KV transfer over XGMI to be correct, and faster than RDMA loopback through the NICs for the copy shapes that inference engines actually send.

Today neither holds:

Correctness. Sub-allocated device buffers (any caching allocator, e.g. PyTorch) read and write the wrong bytes over HIP IPC (#3546). Deployments therefore set MC_DISABLE_HIP=1, and same-node P/D goes GPU → NIC → switch → NIC → GPU.
Performance. Every request is one hipMemcpyAsync. Until #4953, every submit also blocked until the copy finished. P/D KV transfer is thousands of small, scattered copies per request: KV pages, per-head slices, scales and index pages. These run at 0.1–0.7 GB/s over XGMI, 5–20× slower than the same copies over RDMA loopback.

Work items:

#	Item	Status
1	Correct IPC relocation for sub-allocated buffers (#3546)	#4340, follow-up #4488; alternative #3548. Prerequisite for using XGMI in production.
2	Asynchronous submit: enqueue and return, poll events in `getTransferStatus`	#4953 (ready for review)
3	Batched copy kernel for small copies, plus less host work per request	#5040 (draft, stacked on #4953)
4	Order copies after the caller's work without serializing streams	#5044 (draft, stacked on #4953)
5	Optional high-priority copy streams	#5005 (draft)
6	Same-host Store transfers over XGMI	Open question, see below

Motivation: measured on real P/D shapes

Setup: 8× MI355X with all-to-all XGMI, 8 × 400G ionic NICs (one per GPU), ROCm 7.2. One process per GPU over P2PHANDSHAKE, with one submit thread and at most 4096 requests per batch, as in a typical P/D connector. Each case is one request's incremental KV (about 12K tokens) from one prefill GPU, with decode pages scattered at random. The shapes come from the ATOM P/D connector for GLM-5.2 (CPP4 prefill → DCP4 decode) and MiniMax-M3.1 (PP4 prefill → TP4 decode). The table shows p50 per request, with GB/s per prefill GPU in brackets.

Shape (prefill GPU → each decode GPU)	Topology	RDMA loopback	hip, main	hip, #4953	hip, #4953 + #5040
MLA landing slots, 4 × 8 MiB	1→1	1.01 ms (33)	0.60 ms (56)	0.60 ms (56)	0.60 ms (56)
	1→4	3.52 ms (38)	2.40 ms (56)	0.64 ms (206)	0.64 ms (202)
DSA index pages, 3840 × 2112 B	1→1	2.93 ms (2.8)	38.3 ms (0.2)	11.8 ms (0.7)	0.69 ms (11.8)
	1→4	3.79 ms (8.6)	138 ms (0.2)	43.6 ms (0.7)	2.60 ms (12.5)
Head-sliced PAGE KV, 47,520 copies of 64 B–16 KiB	1→1	19.1 ms (3.8)	524 ms (0.1)	136 ms (0.5)	8.08 ms (8.9)
	1→4	138 ms (2.1)	2068 ms (0.1)	562 ms (0.5)	41.2 ms (7.0)
Block PAGE KV, 7200 copies of 512 B–16 KiB	1→1	4.63 ms (16)	77.2 ms (0.9)	16.9 ms (4.3)	2.13 ms (33)
	1→4	11.6 ms (26)	307 ms (0.9)	79.2 ms (3.6)	5.82 ms (49)
16 × 16 MiB contiguous (reference)	1→4	32.5 ms (33)	16.6 ms (65)	4.67 ms (229)	4.65 ms (224)

What the numbers show:

Large copies are fine once submit is asynchronous. One prefill GPU reaches about 200 GB/s to four decode GPUs, versus about 40 GB/s through its one NIC.
Small copies are host-bound. hipMemcpyAsync costs about 2.4 µs per call. hipPointerGetAttributes, hipSetDevice and hipEventQuery cost 30–50 ns each, so the transport's own bookkeeping was the next limit.
Small copies starve under compute. With a bf16 GEMM saturating the sending GPU, small hipMemcpyAsync copies collapse to 0.03–0.04 GB/s: one request takes 1–12 s.

Design of #5040 (item 3)

When the kernel is used. A device's submission with at least MC_HIP_BATCH_COPY_MIN (default 16) copies of at most 1 MiB runs them as one kernel launch over a (src, dst, len) descriptor array.
- Copies to different targets are interleaved, so the workgroups running at any moment write over several links.
- 16-byte aligned ranges use 16-byte vectors. Unaligned copies go to the kernel only up to 4 KiB.
- Copies over 1 MiB stay on hipMemcpyAsync. The copy engines are as fast for them, faster across several links, and use no CUs.
Build and fallback.
- The kernel is compiled with hiprtc on first use per GPU architecture. libhiprtc is dlopened, so there is no new link dependency and no CMAKE_HIP_ARCHITECTURES list for wheels.
- Any failure falls back to hipMemcpyAsync.
- The kernel is used only in IPC mode with peer access enabled between every pair of devices, because it dereferences peer pointers.
Descriptor slots. There are 32 slots per device, each holding 8192 descriptors. Each launch holds a slot until its event fires. When all slots are busy, submit waits for the oldest launch. That depends only on GPU progress, never on the caller polling.
Host trims.
- Relocation is cached per target buffer, and the device is cached per source allocation. This drops a hipSetDevice per request.
- A completion event is not queried again once one slice saw it fire.
- Together these take submit plus poll from about 300 to 170 ns per request.
Interference with compute on the sending GPU (bf16 GEMM, 1486 TFLOPS alone):
- The cost is about 1.1–1.5% GEMM throughput per GB/s sent: −1.3% at 0.3 GB/s, −3.7% at 2 GB/s, −7.4% at 5 GB/s.
- Capping the grid at 32–64 workgroups did not change this, so the cost appears to come from the memory traffic, not from occupied CUs.
- A memory-bound proxy shows −0.5% at 0.3 GB/s and −9% at 5 GB/s. The old path loses 13–16% there while moving almost nothing.
Correctness.
- Unit tests cover random content, lengths from 1 B to 3 MiB, unaligned ranges, more launches than slots, and READ and WRITE.
- A two-process check covers all 16 prefill→decode pairs: each target process verifies its own memory after scattered 2112 B writes.

Item 4: ordering after the caller's stream

Pool streams are non-blocking, so a copy can start before the kernel that produced its source on the caller's stream has finished. That was also true before #4953.

The obvious fix is to record an event on the null stream and make the pool stream wait on it. That halves fan-out bandwidth: 1→4 with 8 MiB copies drops from 207 to 103 GB/s, measured on the setup above.

Options to evaluate:

an explicit stream or event argument on the submit API, which needs API and Python binding changes;
recording on the caller's current stream instead of the legacy null stream;
leaving ordering to callers, as P/D connectors already do by synchronizing a KV-ready event before submit, and documenting it.

Item 6: Store on the same host

The Store client passes explicit RDMA devices, which disables auto-discovery. Only rdma is installed, so same-host GPU↔GPU Store traffic never uses XGMI.

This matters only once Store segments can live in HBM. USE_VRAM_SEGMENT is CUDA-only today. Store owner segments in DRAM are not reachable over XGMI at all. Item 6 is therefore a question for after items 1–3.

Non-goals

A TENT XGMI transport. TENT can only borrow Hygon hylink today; that is tracked separately.
Device-initiated communication on ROCm (EP/PG, IBGDA).
RDMA-side behavior. With many small WRs and 4 submit threads, the 47,520-copy shape took about 10 s per request over RDMA loopback, and 4P→4D 16 MiB writes fell to 13–16 GB/s with a p99 of 175 ms. This deserves its own issue.

Open questions

Default on or off. Should the batch copy kernel be on by default? It is on in #5040. Are 16 copies and 1 MiB the right defaults on MI300X/MI308X, and not only on MI355X?
Runtime compile. Is runtime compilation with hiprtc acceptable? The alternatives are build-time HIP compilation with an architecture list, or shipping a code object.
Caller-side coalescing. Should TE expose a gather API ("copy these N ranges"), so that callers can avoid one TransferRequest and one task per 64 B–16 KiB piece? The remaining 170 ns per request is generic TE bookkeeping (task, slice, markSuccess), not HIP.
CU and QoS controls. Is the measured interference acceptable for prefill GPUs, or should there also be a rate limit or a CU mask?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Jasen2201 on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5041 ### Changes proposed ## Summary This RFC tracks a set of changes to classic Transfer Engine's `HipTransport` (intra-node XGMI on AMD Instinct). The goal is for same-node P/D KV transfer over XGMI to be correct, and faster than RDMA loopback through the NICs for the copy shapes that inference engines actually send. Today neither holds: - **Correctness.** Sub-allocated device buffers (any caching allocator, e.g. PyTorch) read and write the wrong bytes over HIP IPC (#3546). Deployments therefore set `MC_DISABLE_HIP=1`, and same-node P/D goes GPU → NIC → switch → NIC → GPU. - **Performance.** Every request is one `hipMemcpyAsync`. Until #4953, every submit also blocked until the copy finished. P/D KV transfer is thousands of small, scattered copies per request: KV pages, per-head slices, scales and index pages. These run at 0.1–0.7 GB/s over XGMI, 5–20× slower than the same copies over RDMA loopback. Work items: | # | Item | Status | |---|---|---| | 1 | Correct IPC relocation for sub-allocated buffers (#3546) | #4340, follow-up #4488; alternative #3548. **Prerequisite** for using XGMI in production. | | 2 | Asynchronous submit: enqueue and return, poll events in `getTransferStatus` | #4953 (ready for review) | | 3 | Batched copy kernel for small copies, plus less host work per request | #5040 (draft, stacked on #4953) | | 4 | Order copies after the caller's work without serializing streams | #5044 (draft, stacked on #4953) | | 5 | Optional high-priority copy streams | #5005 (draft) | | 6 | Same-host Store transfers over XGMI | Open question, see below | ## Motivation: measured on real P/D shapes Setup: 8× MI355X with all-to-all XGMI, 8 × 400G ionic NICs (one per GPU), ROCm 7.2. One process per GPU over P2PHANDSHAKE, with one submit thread and at most 4096 requests per batch, as in a typical P/D connector. Each case is one request's incremental KV (about 12K tokens) from one prefill GPU, with decode pages scattered at random. The shapes come from the ATOM P/D connector for GLM-5.2 (CPP4 prefill → DCP4 decode) and MiniMax-M3.1 (PP4 prefill → TP4 decode). The table shows p50 per request, with GB/s per prefill GPU in brackets. | Shape (prefill GPU → each decode GPU) | Topology | RDMA loopback | hip, main | hip, #4953 | hip, #4953 + #5040 | |---|---|---|---|---|---| | MLA landing slots, 4 × 8 MiB | 1→1 | 1.01 ms (33) | 0.60 ms (56) | 0.60 ms (56) | 0.60 ms (56) | | | 1→4 | 3.52 ms (38) | 2.40 ms (56) | 0.64 ms (206) | 0.64 ms (202) | | DSA index pages, 3840 × 2112 B | 1→1 | 2.93 ms (2.8) | 38.3 ms (0.2) | 11.8 ms (0.7) | 0.69 ms (11.8) | | | 1→4 | 3.79 ms (8.6) | 138 ms (0.2) | 43.6 ms (0.7) | 2.60 ms (12.5) | | Head-sliced PAGE KV, 47,520 copies of 64 B–16 KiB | 1→1 | 19.1 ms (3.8) | 524 ms (0.1) | 136 ms (0.5) | 8.08 ms (8.9) | | | 1→4 | 138 ms (2.1) | 2068 ms (0.1) | 562 ms (0.5) | 41.2 ms (7.0) | | Block PAGE KV, 7200 copies of 512 B–16 KiB | 1→1 | 4.63 ms (16) | 77.2 ms (0.9) | 16.9 ms (4.3) | 2.13 ms (33) | | | 1→4 | 11.6 ms (26) | 307 ms (0.9) | 79.2 ms (3.6) | 5.82 ms (49) | | 16 × 16 MiB contiguous (reference) | 1→4 | 32.5 ms (33) | 16.6 ms (65) | 4.67 ms (229) | 4.65 ms (224) | What the numbers show: - **Large copies are fine** once submit is asynchronous. One prefill GPU reaches about 200 GB/s to four decode GPUs, versus about 40 GB/s through its one NIC. - **Small copies are host-bound.** `hipMemcpyAsync` costs about 2.4 µs per call. `hipPointerGetAttributes`, `hipSetDevice` and `hipEventQuery` cost 30–50 ns each, so the transport's own bookkeeping was the next limit. - **Small copies starve under compute.** With a bf16 GEMM saturating the sending GPU, small `hipMemcpyAsync` copies collapse to 0.03–0.04 GB/s: one request takes 1–12 s. ## Design of #5040 (item 3) - **When the kernel is used.** A device's submission with at least `MC_HIP_BATCH_COPY_MIN` (default 16) copies of at most 1 MiB runs them as one kernel launch over a `(src, dst, len)` descriptor array. - Copies to different targets are interleaved, so the workgroups running at any moment write over several links. - 16-byte aligned ranges use 16-byte vectors. Unaligned copies go to the kernel only up to 4 KiB. - Copies over 1 MiB stay on `hipMemcpyAsync`. The copy engines are as fast for them, faster across several links, and use no CUs. - **Build and fallback.** - The kernel is compiled with hiprtc on first use per GPU architecture. `libhiprtc` is `dlopen`ed, so there is no new link dependency and no `CMAKE_HIP_ARCHITECTURES` list for wheels. - Any failure falls back to `hipMemcpyAsync`. - The kernel is used only in IPC mode with peer access enabled between every pair of devices, because it dereferences peer pointers. - **Descriptor slots.** There are 32 slots per device, each holding 8192 descriptors. Each launch holds a slot until its event fires. When all slots are busy, submit waits for the oldest launch. That depends only on GPU progress, never on the caller polling. - **Host trims.** - Relocation is cached per target buffer, and the device is cached per source allocation. This drops a `hipSetDevice` per request. - A completion event is not queried again once one slice saw it fire. - Together these take submit plus poll from about 300 to 170 ns per request. - **Interference with compute on the sending GPU** (bf16 GEMM, 1486 TFLOPS alone): - The cost is about 1.1–1.5% GEMM throughput per GB/s sent: −1.3% at 0.3 GB/s, −3.7% at 2 GB/s, −7.4% at 5 GB/s. - Capping the grid at 32–64 workgroups did not change this, so the cost appears to come from the memory traffic, not from occupied CUs. - A memory-bound proxy shows −0.5% at 0.3 GB/s and −9% at 5 GB/s. The old path loses 13–16% there while moving almost nothing. - **Correctness.** - Unit tests cover random content, lengths from 1 B to 3 MiB, unaligned ranges, more launches than slots, and READ and WRITE. - A two-process check covers all 16 prefill→decode pairs: each target process verifies its own memory after scattered 2112 B writes. ## Item 4: ordering after the caller's stream Pool streams are non-blocking, so a copy can start before the kernel that produced its source on the caller's stream has finished. That was also true before #4953. The obvious fix is to record an event on the null stream and make the pool stream wait on it. That **halves fan-out bandwidth**: 1→4 with 8 MiB copies drops from 207 to 103 GB/s, measured on the setup above. Options to evaluate: - an explicit stream or event argument on the submit API, which needs API and Python binding changes; - recording on the caller's *current* stream instead of the legacy null stream; - leaving ordering to callers, as P/D connectors already do by synchronizing a KV-ready event before submit, and documenting it. ## Item 6: Store on the same host The Store client passes explicit RDMA devices, which disables auto-discovery. Only `rdma` is installed, so same-host GPU↔GPU Store traffic never uses XGMI. This matters only once Store segments can live in HBM. `USE_VRAM_SEGMENT` is CUDA-only today. Store owner segments in DRAM are not reachable over XGMI at all. Item 6 is therefore a question for after items 1–3. ## Non-goals - A TENT XGMI transport. TENT can only borrow Hygon `hylink` today; that is tracked separately. - Device-initiated communication on ROCm (EP/PG, IBGDA). - RDMA-side behavior. With many small WRs and 4 submit threads, the 47,520-copy shape took about 10 s per request over RDMA loopback, and 4P→4D 16 MiB writes fell to 13–16 GB/s with a p99 of 175 ms. This deserves its own issue. ## Open questions 1. **Default on or off.** Should the batch copy kernel be on by default? It is on in #5040. Are 16 copies and 1 MiB the right defaults on MI300X/MI308X, and not only on MI355X? 2. **Runtime compile.** Is runtime compilation with hiprtc acceptable? The alternatives are build-time HIP compilation with an architecture list, or shipping a code object. 3. **Caller-side coalescing.** Should TE expose a gather API ("copy these N ranges"), so that callers can avoid one `TransferRequest` and one task per 64 B–16 KiB piece? The remaining 170 ns per request is generic TE bookkeeping (task, slice, `markSuccess`), not HIP. 4. **CU and QoS controls.** Is the measured interference acceptable for prefill GPUs, or should there also be a rate limit or a CU mask? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi commented

2026-10-29 21:24:45 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 28, 2026):

Thanks for opening this issue, @Jasen2201!

Field	Value
Issue	#5041
GitHub user ID	`38908462`
Reporter	@Jasen2201

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 28, 2026): Thanks for opening this issue, @Jasen2201! | Field | Value | |-------|-------| | **Issue** | #5041 | | **GitHub user ID** | `38908462` | | **Reporter** | @Jasen2201 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.