[GH-ISSUE #4408] [Bug][TENT] CPU-only TCP initiators cannot access a remote CUDA buffer #1108

New issue

Closed

opened 2026-10-29 21:22:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 21:22:58 +00:00

Owner

Copy link

Originally created by @feichai0017 on GitHub (Sep 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4408

A TENT client built with USE_CUDA=OFF cannot READ or WRITE a registered CUDA buffer on a CUDA-enabled peer using standard TCP. Both peers initialize TCP, and openSegment/getSegmentInfo succeed, but even a one-byte request fails before payload transfer with task status 6 and Unable to find registered buffer for request.

Reproduced at 719735896c86b56fabec6cf3e825fb2ea640597a (v0.3.13.post1) and c3fa13ecaf1038df933dcb784ff8ffe026381100, using native mooncake::tent::TransferEngine, not the classic TE compatibility layer.

Upstream-only reproduction

The failure also reproduces with unmodified upstream tebench using --backend=tent --xport_type=tcp --tent_transport_hint=tcp: a CPU-only DRAM initiator accesses a CUDA-enabled VRAM target, with P2P metadata, one thread, 4 KiB blocks, batch 1 and --check_consistency=true. Both sides report only TCP. The initiator exits 1 before producing a statistics row, while the bounded target exits cleanly. check_consistency uses mixed WRITE/READ, so this result is not labeled as an isolated operation test. No external transfer harness, OrbitKV library, wrapper, or Manager executes in this reproduction.

For the submitted regression, build the real upstream tent_tcp_transport_test in CPU-only mode and run:

cmake --build build-cpu --target tent_tcp_transport_test -j4
build-cpu/mooncake-transfer-engine/tent/tests/tent_tcp_transport_test \
  --gtest_filter='TcpTransportCapabilityTest.*'

With the production capability assignment unchanged, the real TCP transport and selector return UNSPEC instead of TCP for host/peer-CUDA. This is a selection-metadata test; it neither creates a fake GPU nor substitutes for hardware transfer validation.

Smaller native API reproducer and controls

On one CUDA-capable Linux host, run two processes linked against separate builds of the same revision: USE_TENT=ON, USE_CUDA=OFF for the initiator and USE_TENT=ON, USE_CUDA=ON for the target. Use real CUDA allocations; no CUDA stub is involved.

1. Initialize both engines with P2P metadata and only standard TCP enabled. Use loopback with distinct RPC ports and a bounded synchronization channel.
2. Target: allocate 129 bytes with cudaMalloc, initialize all bytes, and register the allocation as globally readable/writable.
3. Initiator: allocate/register 129 host bytes, open the target segment, and obtain its buffer base and length with getSegmentInfo.
4. Submit one native READ, then repeat in a fresh run with WRITE: local address host_base + 64, remote address remote_base + 64, length 1, explicit TCP transport hint. Keep both registrations and allocations alive throughout submission/polling.
5. Poll to a terminal state. The request returns FAILED rather than COMPLETED. The reported remote range is valid: one byte strictly inside a 129-byte registered allocation.
6. Change only the initiator to the CUDA-enabled build, keeping its allocation in host memory. Both READ and WRITE now pass full-byte and guard checks over two changing-data rounds.

Observed controls:

Initiator / target	Baseline	Main snapshot
CPU-only host client / CUDA target, loopback READ and WRITE	2 fail	2 fail
CUDA-build host client / same CUDA target, loopback READ and WRITE	2 pass	2 pass
CPU-only host client / CUDA target on another physical host	Same failure	Same failure
CUDA client / CPU-only host target, READ and WRITE, 4 KiB–64 MiB	10 pass	10 pass

All GPU tests use real allocation and device/host copies with byte verification. The failing tests perform no early free, unregister or cancellation. These failures concern route selection, not observed data corruption.

Fixed-build control

Moving only the local-host/peer-GPU capability outside the local-device gate makes the native regression and upstream tebench reproduction pass with the CUDA target unchanged. Additional physical-host native API validation passed 14 separate READ/WRITE cases over 1 byte–64 MiB with three changing-data rounds and guards, plus four host controls. Two deliberate GPU-destination corruptions were detected. This confirms the upstream capability restriction; it is not evidence about RDMA or direct GPU networking.

Cause

Capabilities::dram_to_gpu means local DRAM / peer GPU, regardless of the request opcode. However, TcpTransport::install enables it only when the local platform loader is not CPU. The selector therefore rejects the CPU-only initiator's local-host/remote-CUDA pair.

For standard TCP, the remote endpoint performs the device copy; the CPU-only initiator only copies its own host memory. The local CUDA requirement appears unnecessary for this pair. Local-GPU capabilities should retain their local-device requirements. The generic UNSPEC diagnostic also conflates capability rejection with a missing registration.

Related work checked: #4082 covers same-build CUDA peers; #3973 fixes implicit CUDA contexts; RFC #2863 discusses capability semantics. None supplies this mixed-build route fix. Official issue and open/all PR searches found no exact duplicate on 2026-09-30.

AI assistance: OpenAI Codex assisted with reproduction, source inspection, duplicate search, and this report.

Complete upstream tebench reproduction commands

Use separate CPU-only and CUDA-enabled builds of the same Mooncake revision. No application wrapper or custom transfer harness is needed. The failure was reproduced at c3fa13ecaf1038df933dcb784ff8ffe026381100; the patch changes only the CPU endpoint. Keep the CUDA target binary unchanged for the before/after comparison.

cmake -S . -B build-cpu \
  -DCMAKE_BUILD_TYPE=Release -DUSE_TENT=ON -DUSE_CUDA=OFF \
  -DWITH_TE=ON -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF \
  -DBUILD_BENCHMARK=ON -DBUILD_EXAMPLES=OFF -DBUILD_UNIT_TESTS=OFF \
  -DTENT_METRICS_ENABLED=OFF
cmake --build build-cpu --target tebench -j4

cmake -S . -B build-gpu \
  -DCMAKE_BUILD_TYPE=Release -DUSE_TENT=ON -DUSE_CUDA=ON \
  -DWITH_TE=ON -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF \
  -DBUILD_BENCHMARK=ON -DBUILD_EXAMPLES=OFF -DBUILD_UNIT_TESTS=OFF \
  -DTENT_METRICS_ENABLED=OFF
cmake --build build-gpu --target tebench -j4

For a same-host run, build both on a machine with a working CUDA GPU. For physical-host validation, build/run each endpoint on its own machine and replace loopback with that machine's real interface address. Runtime dependencies must be installed normally. Set these values in the appropriate endpoint's shell:

CPU_BUILD="$PWD/build-cpu"
GPU_BUILD="$PWD/build-gpu"
CPU_BIN="$CPU_BUILD/mooncake-transfer-engine/benchmark/tebench"
GPU_BIN="$GPU_BUILD/mooncake-transfer-engine/benchmark/tebench"
CPU_IP=127.0.0.1
GPU_IP=127.0.0.1
GPU_ID=0
CPU_PORT=19502
GPU_PORT=19501
CPU_LIBS="$CPU_BUILD/mooncake-transfer-engine/tent/src/metrics:$CPU_BUILD/mooncake-common"
GPU_LIBS="$GPU_BUILD/mooncake-transfer-engine/tent/src/metrics:$GPU_BUILD/mooncake-common"
export CPU_IP GPU_IP

The explicit build-library directories avoid conflating this transfer test with the separate tebench build-RPATH issue. Keep library directories separate between endpoints. For IPv6, leave configuration IPs unbracketed and bracket only the target endpoint.

Create one configuration per endpoint. On separate hosts, create tcp-cpu.json on the CPU endpoint and tcp-gpu.json on the CUDA endpoint with the corresponding IP set above:

python3 - <<'PY'
import json
import os
from pathlib import Path

for role in ("cpu", "gpu"):
    transports = {
        name: {"enable": False}
        for name in (
            "hp_tcp", "rdma", "shm", "nvlink", "mnnvl", "gds", "io_uring",
            "bufio", "ub", "ascend", "ascend_direct", "sunrise_link",
            "mpcomm", "hylink", "tpu", "xpu", "nccl"
        )
    }
    transports["tcp"] = {"enable": True, "max_concurrent_tasks": 16}
    config = {
        "rpc_server_hostname": os.environ[role.upper() + "_IP"],
        "rpc_server_threads": 1,
        "metadata_type": "p2p",
        "log_level": "info",
        "enable_runtime_queue": False,
        "enable_progress_worker": False,
        "transports": transports,
    }
    Path("tcp-" + role + ".json").write_text(json.dumps(config, indent=2))
PY

Start the CUDA target in one terminal. It owns a bounded 60-second lifetime; wait for its actual segment/listener log before starting the client in the other terminal.

env -u MC_TENT_RPC_THREADS \
  LD_LIBRARY_PATH="$GPU_LIBS" MC_TENT_CONF="$PWD/tcp-gpu.json" \
  timeout --preserve-status --signal=TERM --kill-after=5s 60s \
  "$GPU_BIN" --backend=tent --xport_type=tcp --tent_transport_hint=tcp \
  --metadata_type=p2p --seg_type=VRAM --rpc_server_port="$GPU_PORT" \
  --local_gpu_id="$GPU_ID" --target_gpu_id="$GPU_ID" \
  --total_buffer_size=134217728 \
  --start_num_threads=1 --max_num_threads=1 --duration=1

Run the CPU-only initiator while the target is alive:

TARGET="$GPU_IP:$GPU_PORT"
case "$GPU_IP" in *:*) TARGET="[$GPU_IP]:$GPU_PORT" ;; esac
env -u MC_TENT_RPC_THREADS \
  LD_LIBRARY_PATH="$CPU_LIBS" MC_TENT_CONF="$PWD/tcp-cpu.json" \
  timeout --preserve-status --signal=TERM --kill-after=5s 15s \
  "$CPU_BIN" --backend=tent --xport_type=tcp --tent_transport_hint=tcp \
  --metadata_type=p2p --seg_type=DRAM --rpc_server_port="$CPU_PORT" \
  --target_seg_name="$TARGET" \
  --local_gpu_id="$GPU_ID" --target_gpu_id="$GPU_ID" \
  --total_buffer_size=134217728 \
  --start_num_threads=1 --max_num_threads=1 --duration=1 \
  --check_consistency=true --op_type=mix \
  --start_block_size=4096 --max_block_size=4096 \
  --start_batch_size=1 --max_batch_size=1

On the unpatched CPU build, the initiator exits 1 without a result row and logs Unable to find registered buffer for request. With the CPU capability fix and the same unchanged CUDA target, it exits 0, produces a positive result row, and passes consistency checking. The target exits cleanly when its bounded lifetime ends. Both logs must show actual TCP transport and the requested ports. These are mixed WRITE/READ correctness checks; the observed timing is not a performance claim. Separate READ/WRITE, one-byte, changing-data and guard validation were performed with a native API harness as supplementary evidence.

Originally created by @feichai0017 on GitHub (Sep 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4408 A TENT client built with `USE_CUDA=OFF` cannot READ or WRITE a registered CUDA buffer on a CUDA-enabled peer using standard TCP. Both peers initialize TCP, and `openSegment`/`getSegmentInfo` succeed, but even a one-byte request fails before payload transfer with task status 6 and `Unable to find registered buffer for request`. Reproduced at `719735896c86b56fabec6cf3e825fb2ea640597a` (v0.3.13.post1) and `c3fa13ecaf1038df933dcb784ff8ffe026381100`, using native `mooncake::tent::TransferEngine`, not the classic TE compatibility layer. ### Upstream-only reproduction The failure also reproduces with **unmodified upstream tebench** using `--backend=tent --xport_type=tcp --tent_transport_hint=tcp`: a CPU-only DRAM initiator accesses a CUDA-enabled VRAM target, with P2P metadata, one thread, 4 KiB blocks, batch 1 and `--check_consistency=true`. Both sides report only TCP. The initiator exits 1 before producing a statistics row, while the bounded target exits cleanly. `check_consistency` uses mixed WRITE/READ, so this result is not labeled as an isolated operation test. No external transfer harness, OrbitKV library, wrapper, or Manager executes in this reproduction. For the submitted regression, build the real upstream `tent_tcp_transport_test` in CPU-only mode and run: ```bash cmake --build build-cpu --target tent_tcp_transport_test -j4 build-cpu/mooncake-transfer-engine/tent/tests/tent_tcp_transport_test \ --gtest_filter='TcpTransportCapabilityTest.*' ``` With the production capability assignment unchanged, the real TCP transport and selector return UNSPEC instead of TCP for host/peer-CUDA. This is a selection-metadata test; it neither creates a fake GPU nor substitutes for hardware transfer validation. ### Smaller native API reproducer and controls On one CUDA-capable Linux host, run two processes linked against separate builds of the same revision: `USE_TENT=ON, USE_CUDA=OFF` for the initiator and `USE_TENT=ON, USE_CUDA=ON` for the target. Use real CUDA allocations; no CUDA stub is involved. 1. Initialize both engines with P2P metadata and only standard TCP enabled. Use loopback with distinct RPC ports and a bounded synchronization channel. 2. Target: allocate 129 bytes with `cudaMalloc`, initialize all bytes, and register the allocation as globally readable/writable. 3. Initiator: allocate/register 129 host bytes, open the target segment, and obtain its buffer base and length with `getSegmentInfo`. 4. Submit one native READ, then repeat in a fresh run with WRITE: local address `host_base + 64`, remote address `remote_base + 64`, length 1, explicit TCP transport hint. Keep both registrations and allocations alive throughout submission/polling. 5. Poll to a terminal state. The request returns FAILED rather than COMPLETED. The reported remote range is valid: one byte strictly inside a 129-byte registered allocation. 6. Change only the initiator to the CUDA-enabled build, keeping its allocation in **host** memory. Both READ and WRITE now pass full-byte and guard checks over two changing-data rounds. Observed controls: | Initiator / target | Baseline | Main snapshot | | --- | --- | --- | | CPU-only host client / CUDA target, loopback READ and WRITE | 2 fail | 2 fail | | CUDA-build host client / same CUDA target, loopback READ and WRITE | 2 pass | 2 pass | | CPU-only host client / CUDA target on another physical host | Same failure | Same failure | | CUDA client / CPU-only host target, READ and WRITE, 4 KiB–64 MiB | 10 pass | 10 pass | All GPU tests use real allocation and device/host copies with byte verification. The failing tests perform no early free, unregister or cancellation. These failures concern route selection, not observed data corruption. ### Fixed-build control Moving only the local-host/peer-GPU capability outside the local-device gate makes the native regression and upstream tebench reproduction pass with the CUDA target unchanged. Additional physical-host native API validation passed 14 separate READ/WRITE cases over 1 byte–64 MiB with three changing-data rounds and guards, plus four host controls. Two deliberate GPU-destination corruptions were detected. This confirms the upstream capability restriction; it is not evidence about RDMA or direct GPU networking. ### Cause [`Capabilities::dram_to_gpu`](https://github.com/kvcache-ai/Mooncake/blob/c3fa13ecaf1038df933dcb784ff8ffe026381100/mooncake-transfer-engine/tent/include/tent/runtime/transport.h#L38) means local DRAM / peer GPU, regardless of the request opcode. However, [`TcpTransport::install`](https://github.com/kvcache-ai/Mooncake/blob/c3fa13ecaf1038df933dcb784ff8ffe026381100/mooncake-transfer-engine/tent/src/transport/tcp/tcp_transport.cpp#L98) enables it only when the local platform loader is not CPU. The selector therefore rejects the CPU-only initiator's local-host/remote-CUDA pair. For standard TCP, the remote endpoint performs the device copy; the CPU-only initiator only copies its own host memory. The local CUDA requirement appears unnecessary for this pair. Local-GPU capabilities should retain their local-device requirements. The generic `UNSPEC` diagnostic also conflates capability rejection with a missing registration. Related work checked: #4082 covers same-build CUDA peers; #3973 fixes implicit CUDA contexts; RFC #2863 discusses capability semantics. None supplies this mixed-build route fix. Official issue and open/all PR searches found no exact duplicate on 2026-09-30. AI assistance: OpenAI Codex assisted with reproduction, source inspection, duplicate search, and this report. <details> <summary>Complete upstream tebench reproduction commands</summary> Use separate CPU-only and CUDA-enabled builds of the same Mooncake revision. No application wrapper or custom transfer harness is needed. The failure was reproduced at `c3fa13ecaf1038df933dcb784ff8ffe026381100`; the patch changes only the CPU endpoint. Keep the CUDA target binary unchanged for the before/after comparison. ```bash cmake -S . -B build-cpu \ -DCMAKE_BUILD_TYPE=Release -DUSE_TENT=ON -DUSE_CUDA=OFF \ -DWITH_TE=ON -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF \ -DBUILD_BENCHMARK=ON -DBUILD_EXAMPLES=OFF -DBUILD_UNIT_TESTS=OFF \ -DTENT_METRICS_ENABLED=OFF cmake --build build-cpu --target tebench -j4 cmake -S . -B build-gpu \ -DCMAKE_BUILD_TYPE=Release -DUSE_TENT=ON -DUSE_CUDA=ON \ -DWITH_TE=ON -DWITH_STORE=OFF -DWITH_STORE_RUST=OFF \ -DBUILD_BENCHMARK=ON -DBUILD_EXAMPLES=OFF -DBUILD_UNIT_TESTS=OFF \ -DTENT_METRICS_ENABLED=OFF cmake --build build-gpu --target tebench -j4 ``` For a same-host run, build both on a machine with a working CUDA GPU. For physical-host validation, build/run each endpoint on its own machine and replace loopback with that machine's real interface address. Runtime dependencies must be installed normally. Set these values in the appropriate endpoint's shell: ```bash CPU_BUILD="$PWD/build-cpu" GPU_BUILD="$PWD/build-gpu" CPU_BIN="$CPU_BUILD/mooncake-transfer-engine/benchmark/tebench" GPU_BIN="$GPU_BUILD/mooncake-transfer-engine/benchmark/tebench" CPU_IP=127.0.0.1 GPU_IP=127.0.0.1 GPU_ID=0 CPU_PORT=19502 GPU_PORT=19501 CPU_LIBS="$CPU_BUILD/mooncake-transfer-engine/tent/src/metrics:$CPU_BUILD/mooncake-common" GPU_LIBS="$GPU_BUILD/mooncake-transfer-engine/tent/src/metrics:$GPU_BUILD/mooncake-common" export CPU_IP GPU_IP ``` The explicit build-library directories avoid conflating this transfer test with the separate tebench build-RPATH issue. Keep library directories separate between endpoints. For IPv6, leave configuration IPs unbracketed and bracket only the target endpoint. Create one configuration per endpoint. On separate hosts, create `tcp-cpu.json` on the CPU endpoint and `tcp-gpu.json` on the CUDA endpoint with the corresponding IP set above: ```bash python3 - <<'PY' import json import os from pathlib import Path for role in ("cpu", "gpu"): transports = { name: {"enable": False} for name in ( "hp_tcp", "rdma", "shm", "nvlink", "mnnvl", "gds", "io_uring", "bufio", "ub", "ascend", "ascend_direct", "sunrise_link", "mpcomm", "hylink", "tpu", "xpu", "nccl" ) } transports["tcp"] = {"enable": True, "max_concurrent_tasks": 16} config = { "rpc_server_hostname": os.environ[role.upper() + "_IP"], "rpc_server_threads": 1, "metadata_type": "p2p", "log_level": "info", "enable_runtime_queue": False, "enable_progress_worker": False, "transports": transports, } Path("tcp-" + role + ".json").write_text(json.dumps(config, indent=2)) PY ``` Start the CUDA target in one terminal. It owns a bounded 60-second lifetime; wait for its actual segment/listener log before starting the client in the other terminal. ```bash env -u MC_TENT_RPC_THREADS \ LD_LIBRARY_PATH="$GPU_LIBS" MC_TENT_CONF="$PWD/tcp-gpu.json" \ timeout --preserve-status --signal=TERM --kill-after=5s 60s \ "$GPU_BIN" --backend=tent --xport_type=tcp --tent_transport_hint=tcp \ --metadata_type=p2p --seg_type=VRAM --rpc_server_port="$GPU_PORT" \ --local_gpu_id="$GPU_ID" --target_gpu_id="$GPU_ID" \ --total_buffer_size=134217728 \ --start_num_threads=1 --max_num_threads=1 --duration=1 ``` Run the CPU-only initiator while the target is alive: ```bash TARGET="$GPU_IP:$GPU_PORT" case "$GPU_IP" in *:*) TARGET="[$GPU_IP]:$GPU_PORT" ;; esac env -u MC_TENT_RPC_THREADS \ LD_LIBRARY_PATH="$CPU_LIBS" MC_TENT_CONF="$PWD/tcp-cpu.json" \ timeout --preserve-status --signal=TERM --kill-after=5s 15s \ "$CPU_BIN" --backend=tent --xport_type=tcp --tent_transport_hint=tcp \ --metadata_type=p2p --seg_type=DRAM --rpc_server_port="$CPU_PORT" \ --target_seg_name="$TARGET" \ --local_gpu_id="$GPU_ID" --target_gpu_id="$GPU_ID" \ --total_buffer_size=134217728 \ --start_num_threads=1 --max_num_threads=1 --duration=1 \ --check_consistency=true --op_type=mix \ --start_block_size=4096 --max_block_size=4096 \ --start_batch_size=1 --max_batch_size=1 ``` On the unpatched CPU build, the initiator exits 1 without a result row and logs `Unable to find registered buffer for request`. With the CPU capability fix and the same unchanged CUDA target, it exits 0, produces a positive result row, and passes consistency checking. The target exits cleanly when its bounded lifetime ends. Both logs must show actual TCP transport and the requested ports. These are mixed WRITE/READ correctness checks; the observed timing is not a performance claim. Separate READ/WRITE, one-byte, changing-data and guard validation were performed with a native API harness as supplementary evidence. </details>

bsakaguchi closed this issue 2026-10-29 21:22:58 +00:00

bsakaguchi commented 2026-10-29 21:22:59 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Sep 30, 2026):

Thanks for opening this issue, @feichai0017!

Field	Value
Issue	#4408
GitHub user ID	132118805
Reporter	@feichai0017

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:5903799817 --> @github-actions[bot] commented on GitHub (Sep 30, 2026): Thanks for opening this issue, @feichai0017! | Field | Value | |-------|-------| | **Issue** | #4408 | | **GitHub user ID** | `132118805` | | **Reporter** | @feichai0017 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue 2026-10-29 21:32:08 +00:00

[PR #1108] [MERGED] [CI] Add label event #2016

bsakaguchi referenced this issue 2026-10-29 21:32:27 +00:00

[PR #1180] [CLOSED] [CI] add sglang e2e integration test #2057

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1108

No description provided.