kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #155] 集成vllm压测出现 Received context async event 和 Process failed for slice，性能不及预期 #76

New issue

Closed

opened 2026-10-29 21:08:48 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:08:48 +00:00

Owner

Originally created by @c-guo16 on GitHub (Mar 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/155

使用官方提供的Mooncake集成vllm的方法，两GPU 1P1D，运行参数：

PYTHONPATH=$(pwd) MOONCAKE_CONFIG_PATH=/code/vllm/mooncake.json VLLM_USE_MODELSCOPE=True               \
    python3 -m vllm.entrypoints.openai.api_server                   \
    --model /code/vllm/DeepSeek-V2-Lite-Chat/                \
    --host 0.0.0.0  --port 8100 --max-model-len 20000               \
    --gpu-memory-utilization 0.8                   \
    --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}'    \
    --max-num-seqs 256 --trust_remote_code

压测参数：

python3 ./bench_serving.py \
  --backend vllm --dataset-name random --model /code/vllm/DeepSeek-V2-Lite-Chat/  \
  --tokenizer ./tokenizer \
  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
  --random-input-len 6000 --random-output-len 1000 --random-range-ratio 1 \
  --request-rate 16 --max-concurrency 16 --num-prompts 80 \
  --base-url http://127.0.0.1/:8000  \
  --host 0.0.0.0 --port 8000

压测过程中，prefill机器出现：

decode机器出现大量报错：

E0319 18:44:57.394790 107576 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f57127ac010, length: 65536, dest_addr: 140535498719248, local_nic: mlx5_bond_1, peer_nic: x.x.x.x:13003@mlx5_bond_1, dest_rkey: 41055, retry_cnt: 0): Work Request Flushed Error

压测性能相比pynccl有较大差距，平均TTFT 7600 vs 2250，平均TPOT 34.16ms vs 29.58ms，吞吐2671 vs 3460 token/s

Originally created by @c-guo16 on GitHub (Mar 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/155 使用官方提供的Mooncake集成vllm的方法，两GPU 1P1D，运行参数： ``` PYTHONPATH=$(pwd) MOONCAKE_CONFIG_PATH=/code/vllm/mooncake.json VLLM_USE_MODELSCOPE=True \ python3 -m vllm.entrypoints.openai.api_server \ --model /code/vllm/DeepSeek-V2-Lite-Chat/ \ --host 0.0.0.0 --port 8100 --max-model-len 20000 \ --gpu-memory-utilization 0.8 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}' \ --max-num-seqs 256 --trust_remote_code ``` 压测参数： ``` python3 ./bench_serving.py \ --backend vllm --dataset-name random --model /code/vllm/DeepSeek-V2-Lite-Chat/ \ --tokenizer ./tokenizer \ --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \ --random-input-len 6000 --random-output-len 1000 --random-range-ratio 1 \ --request-rate 16 --max-concurrency 16 --num-prompts 80 \ --base-url http://127.0.0.1/:8000 \ --host 0.0.0.0 --port 8000 ``` 压测过程中，prefill机器出现： ![Image](https://github.com/user-attachments/assets/b43e577e-408d-4bfa-aacd-5cae1f5448f7) decode机器出现大量报错： ``` E0319 18:44:57.394790 107576 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f57127ac010, length: 65536, dest_addr: 140535498719248, local_nic: mlx5_bond_1, peer_nic: x.x.x.x:13003@mlx5_bond_1, dest_rkey: 41055, retry_cnt: 0): Work Request Flushed Error ``` 压测性能相比pynccl有较大差距，平均TTFT 7600 vs 2250，平均TPOT 34.16ms vs 29.58ms，吞吐2671 vs 3460 token/s

bsakaguchi

2026-10-29 21:08:48 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 21, 2025):

不太确定，但有可能--request-rate 16 --max-concurrency 16 QPS太高可能会出现decode来不及取，buffer满的情况，最后可能就变成decode节点本地重计算prefill了。所以反应在吞吐和tpot上，建议提高kv_buffer_size和降低qps试试？另外冒昧问一下您使用的是rdma还是tcp？

@alogfans worker pool这个报错会是上面这个原因吗？

@ShangmingCai commented on GitHub (Mar 21, 2025): 不太确定，但有可能`--request-rate 16 --max-concurrency 16` QPS太高可能会出现decode来不及取，buffer满的情况，最后可能就变成decode节点本地重计算prefill了。所以反应在吞吐和tpot上，建议提高kv_buffer_size和降低qps试试？另外冒昧问一下您使用的是rdma还是tcp？ @alogfans worker pool这个报错会是上面这个原因吗？

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@betrayyourself commented on GitHub (Apr 2, 2025):

目前来看，上面提到的RDMA报错和我压测过程中遇到的卡死都和内存注册相关，如果全程没有注册过large buffer，则不会出现上图所示的RDMA报错，如果有额外注册large buffer的需求就会发生卡死等问题。此外，使用gdr理论上可以更高效地完成kv传输，为什么方案中还要使用CPU内存进行传输呢？

@betrayyourself commented on GitHub (Apr 2, 2025): 目前来看，上面提到的RDMA报错和我压测过程中遇到的卡死都和内存注册相关，如果全程没有注册过large buffer，则不会出现上图所示的RDMA报错，如果有额外注册large buffer的需求就会发生卡死等问题。此外，使用gdr理论上可以更高效地完成kv传输，为什么方案中还要使用CPU内存进行传输呢？

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 2, 2025):

@betrayyourself 因为vllm现在关于PD分离方案的设计抽象不容易做gdr，我们得去侵入vllm gpucache的内存管理才能实现，目前是不太方便去改的；另外多一层cpu内存的buffer也可以支持除PD分离之外与KVCache相关的更多场景。

我们正在对接sglang，方案是会优先走gdr的，后续可以关注一下。

@ShangmingCai commented on GitHub (Apr 2, 2025): @betrayyourself 因为vllm现在关于PD分离方案的设计抽象不容易做gdr，我们得去侵入vllm gpucache的内存管理才能实现，目前是不太方便去改的；另外多一层cpu内存的buffer也可以支持除PD分离之外与KVCache相关的更多场景。我们正在对接sglang，方案是会优先走gdr的，后续可以关注一下。

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@betrayyourself commented on GitHub (Apr 3, 2025):

@ShangmingCai 您好，gdr相关的问题感谢解答。关于large buffer注册（input token size较大时）导致的网卡Work Request Flushed Error以及Reselect remote NIC的问题，想咨询下目前有遇到过么

@betrayyourself commented on GitHub (Apr 3, 2025): @ShangmingCai 您好，gdr相关的问题感谢解答。关于large buffer注册（input token size较大时）导致的网卡Work Request Flushed Error以及Reselect remote NIC的问题，想咨询下目前有遇到过么

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@fanfanaaaa commented on GitHub (Apr 16, 2025):

同样的问题，放个屁股

@fanfanaaaa commented on GitHub (Apr 16, 2025): 同样的问题，放个屁股

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 21, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 21, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 28, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 28, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.