kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #699] [Bug]: Brainstorm about the NVLink transfer issue #224

New issue

Closed

opened 2026-10-29 21:11:08 +00:00 by bsakaguchi · 12 comments

bsakaguchi commented

2026-10-29 21:11:08 +00:00

Owner

Originally created by @fzyzcjy on GitHub (Jul 31, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/699

Originally assigned to: @alogfans, @ShangmingCai on GitHub.

cc @ShangmingCai

(Make this an issue for easier communication)

My naive brainstorms

Is it possible it is because we use cudaMemcpy on sender, and use TCP to notify receiver, and thus receiver's GPU may not yet see those data
- One very naive fix (or at least POC to verify this) may be, on sender cudaMemcpy, we do a st.global with proper memory order to a signal address in the receiver (just like what we do on DeepEP etc), and on the receiver we probe that address with again correct memory order to know whether the data has been arrived.

Originally created by @fzyzcjy on GitHub (Jul 31, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/699 Originally assigned to: @alogfans, @ShangmingCai on GitHub. cc @ShangmingCai (Make this an issue for easier communication) My naive brainstorms * Is it possible it is because we use cudaMemcpy on sender, and use TCP to notify receiver, and thus receiver's GPU may not yet see those data * One very naive fix (or at least POC to verify this) may be, on sender cudaMemcpy, we do a st.global with proper memory order to a signal address in the receiver (just like what we do on DeepEP etc), and on the receiver we probe that address with again correct memory order to know whether the data has been arrived.

bsakaguchi

2026-10-29 21:11:08 +00:00

closed this issue
added the
stale

bug
labels

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jul 31, 2025):

Let us check. CC @alogfans

@ShangmingCai commented on GitHub (Jul 31, 2025): Let us check. CC @alogfans

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@staryxchen commented on GitHub (Jul 31, 2025):

Hi @fzyzcjy
Can you describe the issue you encountered in more detail? I've been trying to optimize nvlink transport recently, so maybe we can solve your problem together 🙂

@staryxchen commented on GitHub (Jul 31, 2025): Hi @fzyzcjy Can you describe the issue you encountered in more detail? I've been trying to optimize nvlink transport recently, so maybe we can solve your problem together 🙂

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@alogfans commented on GitHub (Jul 31, 2025):

We performed write-then-read operations with up to 96 threads on MNNVL machines, and it's likely that the data is consistent in both C++ and Python micro-benchmarks.

In addition, it's interesting that the accurancy drops more significantly if I remove the gil_release_lock in transferSync(). I think there must be something happened to it.

@alogfans commented on GitHub (Jul 31, 2025): We performed write-then-read operations with up to 96 threads on MNNVL machines, and it's likely that the data is consistent in both C++ and Python micro-benchmarks. In addition, it's interesting that the accurancy drops more significantly if I remove the gil_release_lock in transferSync(). I think there must be something happened to it.

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@alogfans commented on GitHub (Jul 31, 2025):

@fzyzcjy Your idea is reasonable, but I'm puzzled about st.global. Also this problem may be happened in RDMA as well (a write completion doesn't indicate that the data is actually arrived at remote).

@alogfans commented on GitHub (Jul 31, 2025): @fzyzcjy Your idea is reasonable, but I'm puzzled about ` st.global`. Also this problem may be happened in RDMA as well (a write completion doesn't indicate that the data is actually arrived at remote).

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@fzyzcjy commented on GitHub (Jul 31, 2025):

Can you describe the issue you encountered in more detail?

@ShangmingCai has all the details (and I guess @alogfans maybe also?) with full reproducible samples etc, maybe he knows more about mooncake internal cluster than me

it's likely that the data is consistent

Yes I also think it should almost true to be consistent...

For st.global just check DeepEP github.com/deepseek-ai/DeepEP@c6faca45c1/csrc/kernels/internode_ll.cu (L232), they use a st.release.sys.global to send the signal

@fzyzcjy commented on GitHub (Jul 31, 2025): > Can you describe the issue you encountered in more detail? @ShangmingCai has all the details (and I guess @alogfans maybe also?) with full reproducible samples etc, maybe he knows more about mooncake internal cluster than me > it's likely that the data is consistent Yes I also think it should almost true to be consistent... For st.global just check DeepEP https://github.com/deepseek-ai/DeepEP/blob/c6faca45c171f099d759e8392772a43da6008bbc/csrc/kernels/internode_ll.cu#L232, they use a `st.release.sys.global` to send the signal

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@alogfans commented on GitHub (Aug 1, 2025):

@fzyzcjy I've wrote a program to simulate the data transfer procedure as follows:

--- Prefill ---          --- Decode ---
transfer_sync(src, dst, len)
zmq.send 
                             zmq.recv
                             validate

Observations:

Only if dst is identical between multiple transfer requests, the decode side may receive corrupted data. In this case, the time diagram looks like this:

--- Prefill ---          --- Decode ---
transfer_sync(src1, dst, len)    <-- V1
zmq.send 
transfer_sync(src2, dst, len)    <-- V2
                             zmq.recv    <-- Data may be mixture of V1 and V2
                             validate

transfer_sync is essentially cudaMemcpy across machines, and NVIDIA controls cache coherence by hardware (at least GPU-to-GPU, which is used by nvlink transport) So just ignore the coherence problem, it's more likely due to unintention overwrite :(
I found that in the SGLang end-to-end test (as you have communicated with us), the first run is usually ok. In the second run, I'm worrying about concurrent write operations triggered by the previous and current run, which applied to the same dest address.

@alogfans commented on GitHub (Aug 1, 2025): @fzyzcjy I've wrote a program to simulate the data transfer procedure as follows: ``` --- Prefill --- --- Decode --- transfer_sync(src, dst, len) zmq.send zmq.recv validate ``` Observations: 1. **Only if** `dst` is identical between multiple transfer requests, the decode side may receive corrupted data. In this case, the time diagram looks like this: ``` --- Prefill --- --- Decode --- transfer_sync(src1, dst, len) <-- V1 zmq.send transfer_sync(src2, dst, len) <-- V2 zmq.recv <-- Data may be mixture of V1 and V2 validate ``` 2. `transfer_sync` is essentially `cudaMemcpy` across machines, and NVIDIA controls cache coherence by hardware (at least GPU-to-GPU, which is used by nvlink transport) So just ignore the coherence problem, it's more likely due to unintention overwrite :( 3. I found that in the SGLang end-to-end test (as you have communicated with us), the first run is usually ok. In the second run, I'm worrying about concurrent write operations triggered by the previous and current run, which applied to the same dest address.

bsakaguchi commented

2026-10-29 21:11:09 +00:00

Author

Owner

@fzyzcjy commented on GitHub (Aug 1, 2025):

I see. Btw curious what raw content do we get when the error happens, and what we get if we simply sleep for a while. (I do not dig into any details about this bug since I heard mooncake will handle it)

@fzyzcjy commented on GitHub (Aug 1, 2025): I see. Btw curious what raw content do we get when the error happens, and what we get if we simply sleep for a while. (I do not dig into any details about this bug since I heard mooncake will handle it)

bsakaguchi commented

2026-10-29 21:11:10 +00:00

Author

Owner

@alogfans commented on GitHub (Aug 1, 2025):

For reference, the test script is attached below. In addition, we should hack writeBytesToBuffer and readBytesFromBuffer functions in Mooncake to support fetching data from vram by an extra copy. @ShangmingCai

scripts.zip

@alogfans commented on GitHub (Aug 1, 2025): For reference, the test script is attached below. In addition, we should hack `writeBytesToBuffer` and `readBytesFromBuffer` functions in Mooncake to support fetching data from vram by an extra copy. @ShangmingCai [scripts.zip](https://github.com/user-attachments/files/21545717/scripts.zip)

bsakaguchi commented

2026-10-29 21:11:10 +00:00

Author

Owner

@alogfans commented on GitHub (Aug 3, 2025):

I have high confidence that the problem can be fixed.
tl; dr: in SGLang's python/sglang/srt/disaggregation/mooncake/transfer_engine.py, insert torch.cuda.synchronize() before and after the call to self.engine.transfer_sync_write.

Reason: When PyTorch writes something to the numpy array, the change may not immediately applied to GPU device. So the transferred data may not be consistent, especially when there are many threads.

@alogfans commented on GitHub (Aug 3, 2025): I have high confidence that the problem can be fixed. tl; dr: in SGLang's `python/sglang/srt/disaggregation/mooncake/transfer_engine.py`, insert `torch.cuda.synchronize()` before **and** after the call to `self.engine.transfer_sync_write`. Reason: When PyTorch writes something to the numpy array, the change may not immediately applied to GPU device. So the transferred data may not be consistent, especially when there are many threads.

bsakaguchi commented

2026-10-29 21:11:10 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Sep 4, 2025):

Temporarily fixed by https://github.com/sgl-project/sglang/pull/9261. But the synchronization problem with small data NVLink transfer remains to be addressed.

@ShangmingCai commented on GitHub (Sep 4, 2025): Temporarily fixed by https://github.com/sgl-project/sglang/pull/9261. But the synchronization problem with small data NVLink transfer remains to be addressed.

bsakaguchi commented

2026-10-29 21:11:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 29, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 29, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:11:10 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jun 29, 2026):

Closed as addressed by the bootstrap room ID check.

@ShangmingCai commented on GitHub (Jun 29, 2026): Closed as addressed by the bootstrap room ID check.

bsakaguchi referenced this issue

2026-10-29 21:27:10 +00:00

[PR #224] [CLOSED] Test 2 28 #1407