kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3428] [Bug]: Mooncake crashes under high concurrency (512 vs 64) due to suspected stack corruption in store.batch_* functions #875

New issue

Closed

opened 2026-10-29 21:20:25 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:20:25 +00:00

Owner

Originally created by @marxpengdc-crypto on GitHub (Aug 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3428

Bug Report

Description
The Mooncake service crashes when the concurrent load increases from 64 to 512. The crash is suspected to be caused by the C++ functions store.batch_is_exist, store.batch_put_from_multi_buffers, and store.batch_is_exist. The stack trace shows an invalid address 0xffffffffffffffff, which is a strong indicator of stack corruption (e.g., a buffer overflow)

The main bug is

(ApiServer_7 pid=172807) INFO: 127.0.0.1:44928 - "GET /metrics HTTP/1.1" 200 OK
!!!!!!! Segfault encountered !!!!!!!
File "", line 0, in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_append(char const*, unsigned long)
File "", line 0, in _PyObject_MakeTpCall
File "", line 0, in _PyEval_EvalFrameDefault
File "", line 0, in 0xffffffffffffffff

vllm version：v0.24.0
mooncake-transfer-engine-cuda13 0.3.11.post1

MOONCAKE_CONFIG_PATH=/tmp/mc_dp8_ssd_v2/mooncake_config.json
PYTHONHASHSEED=0
FLASHINFER_DISABLE_VERSION_CHECK=1
vllm serve deepseek-ai/DeepSeek-V4-Flash
--served-model-name deepseek-v4 --host 0.0.0.0 --port 8000
--tensor-parallel-size 1 --data-parallel-size 8 --data-parallel-size-local 8
--enable-expert-parallel --enable-ep-weight-filter
--all2all-backend deepep_high_throughput --kv-cache-dtype fp8
--gpu-memory-utilization 0.70 --enable-prefix-caching --enforce-eager
--max-model-len 256000 --max-num-seqs 16 --max-num-batched-tokens 8192
--enable-chunked-prefill --enable-cumem-allocator
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"cache_prefix":"mc_dp8_ssd_v2"}}'

mooncake_master
--rpc_port=50051 --rpc_address=0.0.0.0
--enable_offload=true --offload_on_evict=true --promotion_on_hit=true
--enable_disk_eviction=true --allocation_strategy=ssd_free_ratio_first
--root_fs_dir=/tmp/mooncake_offload --quota_bytes=536870912000

aiperf profile
--model deepseek-v4
-deepseek-ai/DeepSeek-V4-Flash
--url http://127.0.0.1:9001 --endpoint-type chat
--streaming --use-server-token-count
--custom-dataset-type multi_turn
--input-file multi_turn.jsonl
--concurrency 512 --workers-max 256 --benchmark-duration 600
--connection-reuse-strategy sticky-user-sessions
--wait-for-model-timeout 0 --export-level raw
--artifact-dir /data-cfs/aiperf_runs/mt512_256k_dp8_mooncake_ssd_v2
--profile-export-prefix mt512_256k_dp8_mooncake_ssd_v2 --ui none

Originally created by @marxpengdc-crypto on GitHub (Aug 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3428 ### Bug Report Description The Mooncake service crashes when the concurrent load increases from 64 to 512. The crash is suspected to be caused by the C++ functions store.batch_is_exist, store.batch_put_from_multi_buffers, and store.batch_is_exist. The stack trace shows an invalid address 0xffffffffffffffff, which is a strong indicator of stack corruption (e.g., a buffer overflow) The main bug is (ApiServer_7 pid=172807) INFO: 127.0.0.1:44928 - "GET /metrics HTTP/1.1" 200 OK !!!!!!! Segfault encountered !!!!!!! File "<unknown>", line 0, in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) File "<unknown>", line 0, in _PyObject_MakeTpCall File "<unknown>", line 0, in _PyEval_EvalFrameDefault File "<unknown>", line 0, in 0xffffffffffffffff vllm version：v0.24.0 mooncake-transfer-engine-cuda13 0.3.11.post1 MOONCAKE_CONFIG_PATH=/tmp/mc_dp8_ssd_v2/mooncake_config.json \ PYTHONHASHSEED=0 \ FLASHINFER_DISABLE_VERSION_CHECK=1 \ vllm serve deepseek-ai/DeepSeek-V4-Flash \ --served-model-name deepseek-v4 --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 1 --data-parallel-size 8 --data-parallel-size-local 8 \ --enable-expert-parallel --enable-ep-weight-filter \ --all2all-backend deepep_high_throughput --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.70 --enable-prefix-caching --enforce-eager \ --max-model-len 256000 --max-num-seqs 16 --max-num-batched-tokens 8192 \ --enable-chunked-prefill --enable-cumem-allocator \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"cache_prefix":"mc_dp8_ssd_v2"}}' mooncake_master \ --rpc_port=50051 --rpc_address=0.0.0.0 \ --enable_offload=true --offload_on_evict=true --promotion_on_hit=true \ --enable_disk_eviction=true --allocation_strategy=ssd_free_ratio_first \ --root_fs_dir=/tmp/mooncake_offload --quota_bytes=536870912000 aiperf profile \ --model deepseek-v4 \ -deepseek-ai/DeepSeek-V4-Flash \ --url http://127.0.0.1:9001 --endpoint-type chat \ --streaming --use-server-token-count \ --custom-dataset-type multi_turn \ --input-file multi_turn.jsonl \ --concurrency 512 --workers-max 256 --benchmark-duration 600 \ --connection-reuse-strategy sticky-user-sessions \ --wait-for-model-timeout 0 --export-level raw \ --artifact-dir /data-cfs/aiperf_runs/mt512_256k_dp8_mooncake_ssd_v2 \ --profile-export-prefix mt512_256k_dp8_mooncake_ssd_v2 --ui none [vllm.log](https://github.com/user-attachments/files/31031066/vllm.log) [client.log](https://github.com/user-attachments/files/31031081/client.log) [master.log](https://github.com/user-attachments/files/31031552/master.log) [mooncake_config.json](https://github.com/user-attachments/files/31031096/mooncake_config.json) [mooncake_metrics.log](https://github.com/user-attachments/files/31031104/mooncake_metrics.log)

bsakaguchi

2026-10-29 21:20:25 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:20:26 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 13, 2026):

Thanks for opening this issue, @marxpengdc-crypto!

Field	Value
Issue	#3428
GitHub user ID	`277691891`
Reporter	@marxpengdc-crypto

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 13, 2026): Thanks for opening this issue, @marxpengdc-crypto! | Field | Value | |-------|-------| | **Issue** | #3428 | | **GitHub user ID** | `277691891` | | **Reporter** | @marxpengdc-crypto | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:20:26 +00:00

Author

Owner

@LujhCoconut commented on GitHub (Aug 14, 2026):

I went through the attached logs.

First, I noticed in vllm.log:

register_buffer failed for addr 0x6672000000 len 38111549760: -600
rdma_context.cpp:521] Failed to register memory 0x6672000000: Bad address [14]

This looks RDMA-MR-related. From my experience it's worth checking whether nvidia-peermem is loaded (lsmod | grep nvidia_peermem), or whether the CUDA driver matches the mooncake-transfer-engine-cuda13 wheel.

Second, I don't see any actual data reads/writes in the logs — Mem Storage and SSD Storage stay at 0 B, and the connector's prefix-cache counters are requests=0, queries=0, hits=0. So the Mooncake data path seems idle at crash time.

What I'm confused about is whether this same deployment actually runs fine at 64 concurrency. The register_buffer failed -600 and Bad address [14] errors are emitted at startup, independent of load — if missing nvidia-peermem were the cause, they should appear at 64 concurrency too. Confirmation of the 64-concurrency behavior would help narrow this down.

@LujhCoconut commented on GitHub (Aug 14, 2026): I went through the attached logs. First, I noticed in `vllm.log`: ``` register_buffer failed for addr 0x6672000000 len 38111549760: -600 rdma_context.cpp:521] Failed to register memory 0x6672000000: Bad address [14] ``` This looks RDMA-MR-related. From my experience it's worth checking whether `nvidia-peermem` is loaded (`lsmod | grep nvidia_peermem`), or whether the CUDA driver matches the `mooncake-transfer-engine-cuda13` wheel. Second, I don't see any actual data reads/writes in the logs — `Mem Storage` and `SSD Storage` stay at 0 B, and the connector's prefix-cache counters are `requests=0, queries=0, hits=0`. So the Mooncake data path seems idle at crash time. What I'm confused about is whether this same deployment actually runs fine at 64 concurrency. The `register_buffer failed -600` and `Bad address [14]` errors are emitted at startup, independent of load — if missing `nvidia-peermem` were the cause, they should appear at 64 concurrency too. Confirmation of the 64-concurrency behavior would help narrow this down.

bsakaguchi commented

2026-10-29 21:20:27 +00:00

Author

Owner

@whn09 commented on GitHub (Aug 14, 2026):

This was routed to me as the EFA transport code owner, but this deployment does not use the EFA transport, so I'm not the right owner for it.

From the attachments:

mooncake_config.json sets "protocol": "rdma" and "device_name": "mlx5_0", and the logs go through rdma_transport.cpp / rdma_context.cpp (Found 1 HCAs, RDMA device: mlx5_0, GID_Index 3).
mooncake-transfer-engine-cuda13 0.3.11.post1 is the generic PyPI CUDA 13 wheel, built with USE_EFA=OFF, so efa_transport isn't compiled into it at all.
There is no efa / libfabric reference anywhere in vllm.log.

So this belongs to the RDMA transport + mooncake-store owners rather than to the EFA path: cc @alogfans @ykwd for reassignment.

Two notes that may save the next person some time:

The mooncake-transfer-engine version in the report doesn't match the binary that produced these logs. PLOG(ERROR) << "Failed to register memory " is at rdma_context.cpp:346 in v0.3.11.post1 and at :588 in v0.3.12.post1, but the log says rdma_context.cpp:521 — so the loaded engine.so is some intermediate build. @marxpengdc-crypto, could you post the exact commit SHA (or pip show -f / the wheel filename) for the module that is actually loaded?
I did read through all four attachments while triaging. I have a log analysis of what the store client and master were doing at crash time that argues against the store.batch_* hypothesis in the issue title; happy to post it here if the new assignee wants it, just ping me.

@whn09 commented on GitHub (Aug 14, 2026): This was routed to me as the EFA transport code owner, but this deployment does not use the EFA transport, so I'm not the right owner for it. From the attachments: - `mooncake_config.json` sets `"protocol": "rdma"` and `"device_name": "mlx5_0"`, and the logs go through `rdma_transport.cpp` / `rdma_context.cpp` (`Found 1 HCAs`, `RDMA device: mlx5_0, GID_Index 3`). - `mooncake-transfer-engine-cuda13 0.3.11.post1` is the generic PyPI CUDA 13 wheel, built with `USE_EFA=OFF`, so `efa_transport` isn't compiled into it at all. - There is no `efa` / `libfabric` reference anywhere in `vllm.log`. So this belongs to the RDMA transport + mooncake-store owners rather than to the EFA path: cc @alogfans @ykwd for reassignment. Two notes that may save the next person some time: 1. The `mooncake-transfer-engine` version in the report doesn't match the binary that produced these logs. `PLOG(ERROR) << "Failed to register memory "` is at `rdma_context.cpp:346` in `v0.3.11.post1` and at `:588` in `v0.3.12.post1`, but the log says `rdma_context.cpp:521` — so the loaded `engine.so` is some intermediate build. @marxpengdc-crypto, could you post the exact commit SHA (or `pip show -f` / the wheel filename) for the module that is actually loaded? 2. I did read through all four attachments while triaging. I have a log analysis of what the store client and master were doing at crash time that argues against the `store.batch_*` hypothesis in the issue title; happy to post it here if the new assignee wants it, just ping me.

bsakaguchi commented

2026-10-29 21:20:27 +00:00

Author

Owner

@ykwd commented on GitHub (Aug 14, 2026):

@whn09 Sorry, I accidentally assigned you the wrong issue. Could you take a look at this one when you have a chance? https://github.com/kvcache-ai/Mooncake/issues/3434

@ykwd commented on GitHub (Aug 14, 2026): @whn09 Sorry, I accidentally assigned you the wrong issue. Could you take a look at this one when you have a chance? [https://github.com/kvcache-ai/Mooncake/issues/3434](https://github.com/kvcache-ai/Mooncake/issues/3434)

bsakaguchi commented

2026-10-29 21:20:27 +00:00

Author

Owner

@marxpengdc-crypto commented on GitHub (Aug 17, 2026):

I went through the attached logs.

First, I noticed in vllm.log:
register_buffer failed for addr 0x6672000000 len 38111549760: -600
rdma_context.cpp:521] Failed to register memory 0x6672000000: Bad address [14]
This looks RDMA-MR-related. From my experience it's worth checking whether nvidia-peermem is loaded (lsmod | grep nvidia_peermem), or whether the CUDA driver matches the mooncake-transfer-engine-cuda13 wheel.

Second, I don't see any actual data reads/writes in the logs — Mem Storage and SSD Storage stay at 0 B, and the connector's prefix-cache counters are requests=0, queries=0, hits=0. So the Mooncake data path seems idle at crash time.

What I'm confused about is whether this same deployment actually runs fine at 64 concurrency. The register_buffer failed -600 and Bad address [14] errors are emitted at startup, independent of load — if missing nvidia-peermem were the cause, they should appear at 64 concurrency too. Confirmation of the 64-concurrency behavior would help narrow this down.

After modprobe nvidia-peermem we fix the problem thanks

@marxpengdc-crypto commented on GitHub (Aug 17, 2026): > I went through the attached logs. > > First, I noticed in `vllm.log`: > > ``` > register_buffer failed for addr 0x6672000000 len 38111549760: -600 > rdma_context.cpp:521] Failed to register memory 0x6672000000: Bad address [14] > ``` > > This looks RDMA-MR-related. From my experience it's worth checking whether `nvidia-peermem` is loaded (`lsmod | grep nvidia_peermem`), or whether the CUDA driver matches the `mooncake-transfer-engine-cuda13` wheel. > > Second, I don't see any actual data reads/writes in the logs — `Mem Storage` and `SSD Storage` stay at 0 B, and the connector's prefix-cache counters are `requests=0, queries=0, hits=0`. So the Mooncake data path seems idle at crash time. > > What I'm confused about is whether this same deployment actually runs fine at 64 concurrency. The `register_buffer failed -600` and `Bad address [14]` errors are emitted at startup, independent of load — if missing `nvidia-peermem` were the cause, they should appear at 64 concurrency too. Confirmation of the 64-concurrency behavior would help narrow this down. After modprobe nvidia-peermem we fix the problem thanks

bsakaguchi referenced this issue

2026-10-29 21:31:04 +00:00

[PR #875] [MERGED] feat(store): Add multi threading handle page fault during segment allocation #1880