kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #2621] [Bug]: SEGMENT_NOT_FOUND, whether to bind the pin thread to an idle core #688

New issue
Open
opened 2026-10-29 21:18:11 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-29 21:18:11 +00:00
Owner
Copy link

Originally created by @huangdong2022 on GitHub (Jun 25, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2621

Originally assigned to: @LujhCoconut, @ascend-direct-dev on GitHub.

Bug Report

Test Scenario:

qwen3.5 397B A2 machine, single server with 8 GPUs, enabled mooncake SSD. Server reports an error: "OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND."
mooncake version : v0.3.11

vllm script

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"
export HCCL_IF_IP=XX.XX.XX.XXX
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_BUFFSIZE=1024

export OMP_NUM_THREADS=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export TASK_QUEUE_ENABLE=1

export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

# mooncake
export PYTHONHASHSEED=0
export HCCL_INTRA_ROCE_ENABLE=1
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH}
export MOONCAKE_CONFIG_PATH="/home/XXX/vllm_server/mooncake-ssd.json"
export ACL_OP_INIT_MODE=1

vllm serve /mnt/nfs/weight/Qwen3.5-397B-A17B-w4a8-mtp \
    --served-model-name "qwen35" \
    --host 0.0.0.0 \
    --port 10010 \
    --data-parallel-size 1 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --max-model-len 133120 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.95 \
    --no-enable-prefix-caching \
    --compilation-config '{"cudagraph_capture_sizes":[1,4,8,12,16,24,32,48,56,64,72,84,96,108,112,128,160,172,196,200,212,232,256], "cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
    --trust-remote-code \
    --async-scheduling \
    --allowed-local-media-path / \
    --quantization ascend \
    --mm-processor-cache-gb 0 \
    --additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}' \
    --no-disable-hybrid-kv-cache-manager \
    --kv-transfer-config \
    '{
    "kv_connector": "AscendStoreConnector",
    "kv_role": "kv_both",
    "kv_load_failure_policy": "fail",
    "kv_connector_extra_config": {
        "lookup_rpc_port":"1",
        "backend": "mooncake"
    }
}' \
2>&1 | tee log0622.log

mooncake-ssd.json

{
  "metadata_server": "P2PHANDSHAKE",
  "protocol": "ascend",
  "device_name": "",
  "master_server_address": "XX.XX.XX.XXX:50088",
  "global_segment_size": "10GB",
  "use_ascend_direct": true,
  "enable_ssd_offload": true,
  "ssd_offload_path": "/home/XXX/mooncake_offload"
}

Error Log

(Worker_TP5_EP5 pid=205995) INFO 06-23 17:39:36 [cpu_binding.py:584] NPU5(PCI 0000:02:00.0): sq_send_trigger_irq IRQ_ID=1563 -> CPU24, cq_update_irq IRQ_ID=1564 -> CPU25
E20260623 17:39:36.985963 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND
E20260623 17:39:36.986027 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND
(Worker_TP4_EP4 pid=205952) INFO 06-23 17:39:37 [cpu_binding.py:584] NPU4(PCI 0000:01:00.0): sq_send_trigger_irq IRQ_ID=1307 -> CPU0, cq_update_irq IRQ_ID=1308 -> CPU1
(Worker_TP7_EP7 pid=206082) INFO 06-23 17:39:40 [cpu_binding.py:584] NPU7(PCI 0000:42:00.0): sq_send_trigger_irq IRQ_ID=2075 -> CPU72, cq_update_irq IRQ_ID=2076 -> CPU73
(Worker_TP1_EP1 pid=205825) INFO 06-23 17:39:42 [cpu_binding.py:584] NPU1(PCI 0000:c2:00.0): sq_send_trigger_irq IRQ_ID=3099 -> CPU168, cq_update_irq IRQ_ID=3100 -> CPU169
(Worker_TP2_EP2 pid=205866) INFO 06-23 17:39:42 [cpu_binding.py:584] NPU2(PCI 0000:81:00.0): sq_send_trigger_irq IRQ_ID=2331 -> CPU96, cq_update_irq IRQ_ID=2332 -> CPU97
(Worker_TP0_EP0 pid=205806) INFO 06-23 17:39:43 [cpu_binding.py:584] NPU0(PCI 0000:c1:00.0): sq_send_trigger_irq IRQ_ID=2843 -> CPU144, cq_update_irq IRQ_ID=2844 -> CPU145
(Worker_TP3_EP3 pid=205909) INFO 06-23 17:39:46 [cpu_binding.py:584] NPU3(PCI 0000:82:00.0): sq_send_trigger_irq IRQ_ID=2587 -> CPU120, cq_update_irq IRQ_ID=2588 -> CPU121
E20260623 17:39:46.986565 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND
E20260623 17:39:46.986617 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND
(Worker_TP6_EP6 pid=206038) INFO 06-23 17:39:48 [cpu_binding.py:584] NPU6(PCI 0000:41:00.0): sq_send_trigger_irq IRQ_ID=1819 -> CPU48, cq_update_irq IRQ_ID=1820 -> CPU49
(EngineCore pid=205782) INFO 06-23 17:39:48 [core.py:302] init engine (profile, create kv cache, warmup model) took 201.23 s (compilation: 117.02 s)
(EngineCore pid=205782) INFO 06-23 17:39:50 [factory.py:62] Creating v1 connector with name: AscendStoreConnector and engine_id: 14bfa59e-0b65-480c-9acb-e7e4e461d51b-d9b5ebfebde344e08fd7b14d99a33e58
(EngineCore pid=205782) WARNING 06-23 17:39:50 [base.py:190] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
(EngineCore pid=205782) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=205782) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
E20260623 17:39:56.987149 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND
E20260623 17:39:56.987206 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND

Question

If client logs show OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND, Master has unmounted the rank's LOCAL_DISK segment (usually after client_expired when Ping stops refreshing TTL). SSD offload on that rank stops until the segment is registered again.

Typical trigger (with enable_cpu_binding=true): Mooncake starts Ping during init, then vLLM-Ascend bind_cpus() runs migratepages/IRQ binding; the Ping thread is not pinned and can miss beats under the default client_ttl=10.

Mitigation Notes
Temporary: raise Master TTL e.g. mooncake_master ... --client_ttl=120. Tune to your init/warmup window (often 60–120 is enough). Does not fix the root cause.
Recovery: upgrade Mooncake Versions > v0.3.11 (main branch) can remount LOCAL_DISK and rescan metadata after SEGMENT_NOT_FOUND. This recovers after cleanup; it does not prevent expiry or in-flight request failures while metadata is gone.
Root fix: Mooncake Ping CPU affinity Pin the storage Ping thread to a release/isolated CPU (Mooncake-side change). Optional vLLM-Ascend cooperation to pass the release CPU per rank.

The Recovery solution also cannot completely eliminate the risk of LOCAL_DISK being cleared by the master. If a request is currently fetching keys from SSDs, and then the pin thread fails to be scheduled within a certain period, LOCAL_DISK will still be released. If the model cannot enable recomputation, the request will fail at this point. Later, we will consider whether to bind the pin thread to an idle core.

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @huangdong2022 on GitHub (Jun 25, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2621 Originally assigned to: @LujhCoconut, @ascend-direct-dev on GitHub. ### Bug Report ### Test Scenario: qwen3.5 397B A2 machine, single server with 8 GPUs, enabled mooncake SSD. Server reports an error: "OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND." mooncake version : v0.3.11 ### vllm script ```shell export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True" export HCCL_IF_IP=XX.XX.XX.XXX export HCCL_OP_EXPANSION_MODE="AIV" export HCCL_BUFFSIZE=1024 export OMP_NUM_THREADS=1 export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD export TASK_QUEUE_ENABLE=1 export VLLM_ASCEND_ENABLE_FUSED_MC2=1 export VLLM_ASCEND_ENABLE_FLASHCOMM1=1 # mooncake export PYTHONHASHSEED=0 export HCCL_INTRA_ROCE_ENABLE=1 #export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH} export MOONCAKE_CONFIG_PATH="/home/XXX/vllm_server/mooncake-ssd.json" export ACL_OP_INIT_MODE=1 vllm serve /mnt/nfs/weight/Qwen3.5-397B-A17B-w4a8-mtp \ --served-model-name "qwen35" \ --host 0.0.0.0 \ --port 10010 \ --data-parallel-size 1 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --max-model-len 133120 \ --max-num-batched-tokens 16384 \ --max-num-seqs 128 \ --gpu-memory-utilization 0.95 \ --no-enable-prefix-caching \ --compilation-config '{"cudagraph_capture_sizes":[1,4,8,12,16,24,32,48,56,64,72,84,96,108,112,128,160,172,196,200,212,232,256], "cudagraph_mode":"FULL_DECODE_ONLY"}' \ --speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \ --trust-remote-code \ --async-scheduling \ --allowed-local-media-path / \ --quantization ascend \ --mm-processor-cache-gb 0 \ --additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}' \ --no-disable-hybrid-kv-cache-manager \ --kv-transfer-config \ '{ "kv_connector": "AscendStoreConnector", "kv_role": "kv_both", "kv_load_failure_policy": "fail", "kv_connector_extra_config": { "lookup_rpc_port":"1", "backend": "mooncake" } }' \ 2>&1 | tee log0622.log ``` mooncake-ssd.json ```text { "metadata_server": "P2PHANDSHAKE", "protocol": "ascend", "device_name": "", "master_server_address": "XX.XX.XX.XXX:50088", "global_segment_size": "10GB", "use_ascend_direct": true, "enable_ssd_offload": true, "ssd_offload_path": "/home/XXX/mooncake_offload" } ``` ### Error Log ```text (Worker_TP5_EP5 pid=205995) INFO 06-23 17:39:36 [cpu_binding.py:584] NPU5(PCI 0000:02:00.0): sq_send_trigger_irq IRQ_ID=1563 -> CPU24, cq_update_irq IRQ_ID=1564 -> CPU25 E20260623 17:39:36.985963 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND E20260623 17:39:36.986027 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND (Worker_TP4_EP4 pid=205952) INFO 06-23 17:39:37 [cpu_binding.py:584] NPU4(PCI 0000:01:00.0): sq_send_trigger_irq IRQ_ID=1307 -> CPU0, cq_update_irq IRQ_ID=1308 -> CPU1 (Worker_TP7_EP7 pid=206082) INFO 06-23 17:39:40 [cpu_binding.py:584] NPU7(PCI 0000:42:00.0): sq_send_trigger_irq IRQ_ID=2075 -> CPU72, cq_update_irq IRQ_ID=2076 -> CPU73 (Worker_TP1_EP1 pid=205825) INFO 06-23 17:39:42 [cpu_binding.py:584] NPU1(PCI 0000:c2:00.0): sq_send_trigger_irq IRQ_ID=3099 -> CPU168, cq_update_irq IRQ_ID=3100 -> CPU169 (Worker_TP2_EP2 pid=205866) INFO 06-23 17:39:42 [cpu_binding.py:584] NPU2(PCI 0000:81:00.0): sq_send_trigger_irq IRQ_ID=2331 -> CPU96, cq_update_irq IRQ_ID=2332 -> CPU97 (Worker_TP0_EP0 pid=205806) INFO 06-23 17:39:43 [cpu_binding.py:584] NPU0(PCI 0000:c1:00.0): sq_send_trigger_irq IRQ_ID=2843 -> CPU144, cq_update_irq IRQ_ID=2844 -> CPU145 (Worker_TP3_EP3 pid=205909) INFO 06-23 17:39:46 [cpu_binding.py:584] NPU3(PCI 0000:82:00.0): sq_send_trigger_irq IRQ_ID=2587 -> CPU120, cq_update_irq IRQ_ID=2588 -> CPU121 E20260623 17:39:46.986565 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND E20260623 17:39:46.986617 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND (Worker_TP6_EP6 pid=206038) INFO 06-23 17:39:48 [cpu_binding.py:584] NPU6(PCI 0000:41:00.0): sq_send_trigger_irq IRQ_ID=1819 -> CPU48, cq_update_irq IRQ_ID=1820 -> CPU49 (EngineCore pid=205782) INFO 06-23 17:39:48 [core.py:302] init engine (profile, create kv cache, warmup model) took 201.23 s (compilation: 117.02 s) (EngineCore pid=205782) INFO 06-23 17:39:50 [factory.py:62] Creating v1 connector with name: AscendStoreConnector and engine_id: 14bfa59e-0b65-480c-9acb-e7e4e461d51b-d9b5ebfebde344e08fd7b14d99a33e58 (EngineCore pid=205782) WARNING 06-23 17:39:50 [base.py:190] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design. (EngineCore pid=205782) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead. (EngineCore pid=205782) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. E20260623 17:39:56.987149 216570 client_service.cpp:2438] OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND E20260623 17:39:56.987206 216570 file_storage.cpp:510] Failed to send heartbeat with error: SEGMENT_NOT_FOUND ``` ### Question If client logs show `OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND`, Master has unmounted the rank's `LOCAL_DISK` segment (usually after `client_expired` when Ping stops refreshing TTL). SSD offload on that rank stops until the segment is registered again. **Typical trigger (with `enable_cpu_binding=true`):** Mooncake starts Ping during init, then vLLM-Ascend `bind_cpus()` runs `migratepages`/IRQ binding; the Ping thread is not pinned and can miss beats under the default `client_ttl=10`. | Mitigation | Notes | | :--- | :--- | | **Temporary:** raise Master TTL | e.g. `mooncake_master ... --client_ttl=120`. Tune to your init/warmup window (often `60`–`120` is enough). Does not fix the root cause. | | **Recovery:** upgrade Mooncake | Versions **> v0.3.11** (main branch) can remount `LOCAL_DISK` and rescan metadata after `SEGMENT_NOT_FOUND`. This **recovers after** cleanup; it does **not** prevent expiry or in-flight request failures while metadata is gone. | | **Root fix:** Mooncake Ping CPU affinity | Pin the storage Ping thread to a release/isolated CPU (Mooncake-side change). Optional vLLM-Ascend cooperation to pass the release CPU per rank. | The Recovery solution also cannot completely eliminate the risk of LOCAL_DISK being cleared by the master. If a request is currently fetching keys from SSDs, and then the pin thread fails to be scheduled within a certain period, LOCAL_DISK will still be released. If the model cannot enable recomputation, the request will fail at this point. Later, we will consider whether to bind the pin thread to an idle core. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi commented 2026-10-29 21:18:12 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 25, 2026):

Thanks for opening this issue, @huangdong2022!

Field Value
Issue #2621
GitHub user ID 161736910
Reporter @huangdong2022

A maintainer will triage this when possible. To help us respond faster, please include:

  • Mooncake version or commit SHA
  • Environment (OS, CUDA/driver, RDMA stack if relevant)
  • Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4798743289 --> @github-actions[bot] commented on GitHub (Jun 25, 2026): Thanks for opening this issue, @huangdong2022! | Field | Value | |-------|-------| | **Issue** | #2621 | | **GitHub user ID** | `161736910` | | **Reporter** | @huangdong2022 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.
bsakaguchi commented 2026-10-29 21:18:12 +00:00
Author
Owner
Copy link

@LujhCoconut commented on GitHub (Jun 26, 2026):

Thank you for your thorough and insightful analysis. From my perspective, this appears to be a rather challenging problem.

  • While core affinity offers the most straightforward and intuitive solution, it inevitably underutilizes CPU resources.
  • At its core, this seems to be a scheduling priority issue, and elevating the preemption priority of the ping thread could potentially mitigate it. That said, although preemption itself is typically lightweight, if it happens to coincide with the critical path of a kernel launch, it may introduce bubbles into the GPU/NPU pipeline, potentially resulting in non-negligible performance degradation.

Therefore, perhaps making this an optional strategy would be more appropriate?
Happy to discuss further.

<!-- gh-comment-id:4806176828 --> @LujhCoconut commented on GitHub (Jun 26, 2026): Thank you for your thorough and insightful analysis. From my perspective, this appears to be a rather challenging problem. * While core affinity offers the most straightforward and intuitive solution, it inevitably underutilizes CPU resources. * At its core, this seems to be a scheduling priority issue, and elevating the preemption priority of the ping thread could potentially mitigate it. That said, although preemption itself is typically lightweight, if it happens to coincide with the critical path of a kernel launch, it may introduce bubbles into the GPU/NPU pipeline, potentially resulting in non-negligible performance degradation. Therefore, perhaps making this an optional strategy would be more appropriate? Happy to discuss further.
bsakaguchi commented 2026-10-29 21:18:13 +00:00
Author
Owner
Copy link

@ykwd commented on GitHub (Jun 29, 2026):

@LujhCoconut Currently, when a client gets reconnected to master, will it re-mount the ssd segment?

<!-- gh-comment-id:4828515135 --> @ykwd commented on GitHub (Jun 29, 2026): @LujhCoconut Currently, when a client gets reconnected to master, will it re-mount the ssd segment?
bsakaguchi commented 2026-10-29 21:18:13 +00:00
Author
Owner
Copy link

@LujhCoconut commented on GitHub (Jun 29, 2026):

@LujhCoconut Currently, when a client gets reconnected to master, will it re-mount the ssd segment?

Yes. It is introduced in this PR (PR #2077 ). Just note that the SSD remount happens on the next FileStorage heartbeat cycle (default every 10s), not immediately during reconnection itself — so there's a brief recovery window.

<!-- gh-comment-id:4828601488 --> @LujhCoconut commented on GitHub (Jun 29, 2026): > [@LujhCoconut](https://github.com/LujhCoconut) Currently, when a client gets reconnected to master, will it re-mount the ssd segment? Yes. It is introduced in this PR (PR #2077 ). Just note that the SSD remount happens on the next FileStorage heartbeat cycle (default every 10s), not immediately during reconnection itself — so there's a brief recovery window.
bsakaguchi commented 2026-10-29 21:18:13 +00:00
Author
Owner
Copy link

@WANG-WADE commented on GitHub (Aug 6, 2026):

any progress on this issue so far? seem still not fixed...

<!-- gh-comment-id:5202346134 --> @WANG-WADE commented on GitHub (Aug 6, 2026): any progress on this issue so far? seem still not fixed...
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#688
No description provided.