[GH-ISSUE #561] [Question] Why do we need Serialization in KV transfer? #189

New issue

Closed

opened 2026-10-29 21:10:39 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:10:39 +00:00

Owner

Originally created by @hyuenmin-choi on GitHub (Jun 26, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/561

Hello guys!
First of all, thank you guys maintaining this awesome project.
I'm just one of the newcomers who trying to deep dive into this project.

As mentioned in issue #213, the Mooncake store currently uses safetensor_save in the put_impl function to serialize tensors before transmission. I understand that GPUDirect RDMA and layer-by-layer KV cache transfer are not yet fully supported in the vLLM integration.

However, I have a more fundamental question: Why do we need to serialize the tensor at all before sending it over the network?

In send_kv_caches_and_hidden_states() (under vllm/kv_connector/mooncake_store), all the KV caches that need to be transferred are already concatenated into a single tensor before serialization:

Since the data is already contiguous in memory, wouldn’t it be more efficient to directly memcpy from the tensor’s data_ptr() and send that along with the metadata, instead of going through safetensor_save()?

I'm still new to RDMA transfers and distributed KV cache handling, so I’m wondering if there are any technical reasons or constraints (e.g., network protocol limitations, cross-device synchronization, memory alignment, etc.) that prevent this simpler approach.

When I profiled this, the time spent on safetensor_save() actually dominates the send_kv_caches_and_hidden_states() call — it takes more than half of the total time. So I'm curious if this is just a temporary workaround, or if there's a deeper reason for relying on safetensor_save.

If anyone with more context could chime in, I’d really appreciate it!
Thanks again for the great work!

Originally created by @hyuenmin-choi on GitHub (Jun 26, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/561 Hello guys! First of all, thank you guys maintaining this awesome project. I'm just one of the newcomers who trying to deep dive into this project. As mentioned in issue #213, the Mooncake store currently uses safetensor_save in the put_impl function to serialize tensors before transmission. I understand that GPUDirect RDMA and layer-by-layer KV cache transfer are not yet fully supported in the vLLM integration. However, I have a more fundamental question: **Why do we need to serialize the tensor at all before sending it over the network?** In send_kv_caches_and_hidden_states() (under vllm/kv_connector/mooncake_store), all the KV caches that need to be transferred are already concatenated into a single tensor before serialization: <img width="552" alt="Image" src="https://github.com/user-attachments/assets/f75b25ba-b92e-43f9-9a44-0e01315ab9a8" /> **Since the data is already contiguous in memory, wouldn’t it be more efficient to directly memcpy from the tensor’s data_ptr() and send that along with the metadata, instead of going through safetensor_save()?** I'm still new to RDMA transfers and distributed KV cache handling, so I’m wondering if there are any technical reasons or constraints (e.g., network protocol limitations, cross-device synchronization, memory alignment, etc.) that prevent this simpler approach. When I profiled this, the time spent on safetensor_save() actually dominates the send_kv_caches_and_hidden_states() call — it takes more than half of the total time. So I'm curious if this is just a temporary workaround, or if there's a deeper reason for relying on safetensor_save. If anyone with more context could chime in, I’d really appreciate it! Thanks again for the great work!

bsakaguchi

2026-10-29 21:10:39 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 26, 2025):

Yeah, I think this is just a temporary workaround.

The ideal approach would be something like this: First, we make sure the tensor is allocated on registered CPU memory (so it's ready for future RDMA transfers), and then we use the zero-copy API, like put_from.

On the receiving side, we need to create a tensor with exactly the same shape and dtype as the one on the sender’s side, also using registered memory. Then we can use get_into for the zero-copy receive. Maybe we can use tensor.view() for that, but we still need to know the shape and type ahead of time. After that, we can move the tensor back to GPU as needed.

Just some thoughts! Also, since v0 is about to deprecated, maybe we should focus on v1 going forward?

@xiaguan commented on GitHub (Jun 26, 2025): Yeah, I think this is just a temporary workaround. The ideal approach would be something like this: First, we make sure the tensor is allocated on registered CPU memory (so it's ready for future RDMA transfers), and then we use the zero-copy API, like `put_from`. On the receiving side, we need to create a tensor with exactly the same shape and dtype as the one on the sender’s side, also using registered memory. Then we can use `get_into` for the zero-copy receive. Maybe we can use `tensor.view()` for that, but we still need to know the shape and type ahead of time. After that, we can move the tensor back to GPU as needed. Just some thoughts! Also, since v0 is about to deprecated, maybe we should focus on v1 going forward?

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 2, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 2, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 10, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 10, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:56 +00:00

[PR #189] [MERGED] [CI] Refine release workflow to enable publishing to PyPI #1382