kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #221] We found that deleting slices leads to poor performance。delete slice 有性能问题 #94

New issue

Closed

opened 2026-10-29 21:09:11 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:09:11 +00:00

Owner

Originally created by @a4zhangfei on GitHub (Apr 9, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/221

github.com/kvcache-ai/Mooncake@5ed7331871
原本有内存泄漏的问题，后面有PR修复了，升级之后不会内存泄漏了，但是会导致性能变差，（500ms）怀疑是 delete 导致，注释掉下面的就没问题

Originally created by @a4zhangfei on GitHub (Apr 9, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/221 https://github.com/kvcache-ai/Mooncake/commit/5ed7331871cd9c0d146d43e81404796fdea74789 原本有内存泄漏的问题，后面有PR修复了，升级之后不会内存泄漏了，但是会导致性能变差，（500ms）怀疑是 delete 导致，注释掉下面的就没问题 ![Image](https://github.com/user-attachments/assets/5e393eb1-c28c-4524-904c-1a7e5c072fce)

bsakaguchi closed this issue

2026-10-29 21:09:11 +00:00

bsakaguchi commented

2026-10-29 21:09:12 +00:00

Author

Owner

@xiaguan commented on GitHub (Apr 9, 2025):

Is there a reproducible case available? Or, could you share how you identified this performance issue?

@xiaguan commented on GitHub (Apr 9, 2025): Is there a reproducible case available? Or, could you share how you identified this performance issue?

bsakaguchi commented

2026-10-29 21:09:12 +00:00

Author

Owner

@xiaguan commented on GitHub (Apr 9, 2025):

By the way, pr #220 fixed a known long-tail issue, which might resemble the problem you described. You might want to try updating and see if it helps?

@xiaguan commented on GitHub (Apr 9, 2025): By the way, pr #220 fixed a known long-tail issue, which might resemble the problem you described. You might want to try updating and see if it helps?

bsakaguchi commented

2026-10-29 21:09:12 +00:00

Author

Owner

@a4zhangfei commented on GitHub (Apr 9, 2025):

By the way, pr #220 fixed a known long-tail issue, which might resemble the problem you described. You might want to try updating and see if it helps?

@xiaguan
thanks for your reply. However, update to the latest code did not resolve the performance issue.
we use it transfer kv data from prefill instance to decode instance layer-wise.
P/D instance TP4，slice_count: 2w~3w / layer, too many slices cause performance issues with delete.

@a4zhangfei commented on GitHub (Apr 9, 2025): > By the way, pr [#220](https://github.com/kvcache-ai/Mooncake/pull/220) fixed a known long-tail issue, which might resemble the problem you described. You might want to try updating and see if it helps? @xiaguan thanks for your reply. However, update to the latest code did not resolve the performance issue. we use it transfer kv data from prefill instance to decode instance layer-wise. P/D instance TP4，slice_count: 2w~3w / layer, too many slices cause performance issues with delete.

bsakaguchi commented

2026-10-29 21:09:13 +00:00

Author

Owner

@xiaguan commented on GitHub (Apr 9, 2025):

You could also check here github.com/kvcache-ai/Mooncake@b31b8b84b9/mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp (L215C1-L222C75)

@xiaguan commented on GitHub (Apr 9, 2025): You could also check here https://github.com/kvcache-ai/Mooncake/blob/b31b8b84b9f1871a4582a70273f0a6a8626558fd/mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp#L215C1-L222C75

bsakaguchi commented

2026-10-29 21:09:13 +00:00

Author

Owner

@alogfans commented on GitHub (Apr 10, 2025):

@a4zhangfei In the previous patch, the delete operations are internally executed in the transfer engine's worker thread. So we moved them back the call of freeBatchID. We also use a thread-local Slice cache to reduce the call of new/delete.

@alogfans commented on GitHub (Apr 10, 2025): @a4zhangfei In the previous patch, the delete operations are internally executed in the transfer engine's worker thread. So we moved them back the call of freeBatchID. We also use a thread-local Slice cache to reduce the call of new/delete.

bsakaguchi commented

2026-10-29 21:09:13 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 10, 2025):

Closed as fixed.

@ShangmingCai commented on GitHub (Apr 10, 2025): Closed as fixed.

bsakaguchi referenced this issue

2026-10-29 21:26:21 +00:00

[PR #94] [MERGED] [Transfer Engine] add USE_ETCD option to retire the etcd dependency. #1320