[GH-ISSUE #597] [Bug]: Deepseek xPyD error TypeError: cumsum() received an invalid combination of arguments #199

New issue

Closed

opened 2026-10-29 21:10:48 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 21:10:48 +00:00

Owner

Copy link

Originally created by @mugglew on GitHub (Jul 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/597

Bug Report

When i starts the first prefill node it reports error log below:

[2025-07-06 23:33:30 DP6 TP6] Try DeepGEMM JIT Compiling for <gemm_fp8_fp8_bf16_nt> N=7168, K=18432, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run python3 -m sglang.compile_deep_gemm.
100%|██████████| 16/16 [00:09<00:00, 1.77it/s]
[2025-07-06 23:33:41 DP0 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last):
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 127, in forward_thread_func
self.forward_thread_func_()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 162, in forward_thread_func
self.worker.forward_batch_generation(
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 212, in forward_batch_generation
logits_output, can_run_cuda_graph = self.model_runner.forward(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1231, in forward
output = self._forward_raw(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1260, in _forward_raw
ret = self.forward_extend(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1199, in forward_extend
return self.model.forward(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1962, in forward
hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1860, in forward
hidden_states, residual = model_forward_maybe_tbo(
File "/sgl-workspace/sglang/python/sglang/srt/two_batch_overlap.py", line 462, in model_forward_maybe_tbo
return _model_forward_tbo(
File "/sgl-workspace/sglang/python/sglang/srt/two_batch_overlap.py", line 488, in _model_forward_tbo
outputs_arr = execute_overlapped_operations(
File "/sgl-workspace/sglang/python/sglang/srt/operations.py", line 44, in execute_overlapped_operations
executor_a.next()
File "/sgl-workspace/sglang/python/sglang/srt/operations.py", line 84, in next
self._stage_output = op.fn(
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1741, in op_comm_prepare_mlp
self.layer_communicator.prepare_mlp(
File "/sgl-workspace/sglang/python/sglang/srt/layers/communicator.py", line 205, in prepare_mlp
return self._communicate_with_all_reduce_and_layer_norm_fn(
File "/sgl-workspace/sglang/python/sglang/srt/layers/communicator.py", line 395, in _gather_hidden_states_and_residual
dp_gather_partial(hidden_states, local_hidden_states, forward_batch)
File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 269, in dp_gather_partial
_dp_gather(global_tokens, local_tokens, forward_batch, is_partial=True)
File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 232, in _dp_gather
local_start_pos, local_num_tokens = get_dp_local_info(forward_batch)
File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 172, in get_dp_local_info
cumtokens = torch.cumsum(forward_batch.global_num_tokens_gpu, dim=0)
TypeError: cumsum() received an invalid combination of arguments - got (NoneType, dim=int), but expected one of:

(Tensor input, int dim, *, torch.dtype dtype = None, Tensor out = None)
(Tensor input, name dim, *, torch.dtype dtype = None, Tensor out = None)
100%|██████████| 16/16 [00:08<00:00, 1.81it/s]
[2025-07-06 23:33:41] Child process unexpectedly failed with exitcode=131. pid=9808`

my command

MC_TE_METRIC=true \
GLOO_SOCKET_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
SGLANG_TBO_DEBUG=1 \
python3 -m sglang.launch_server \
--model-path /upfs/models/deepseek-ai/DeepSeek-R1-0528 \
--disaggregation-ib-device mlx5_1 \
--disaggregation-mode prefill \
--dist-init-addr 192.168.159.48:5000 \
--nnodes 2 \
--node-rank 1 \
--tp-size 16 \
--dp-size 16 \
--enable-dp-attention \
--page-size 1 \
--host 0.0.0.0 \
--trust-remote-code \
--moe-dense-tp-size 1 \
--enable-dp-lm-head \
--disable-radix-cache \
--watchdog-timeout 1000000 \
--enable-two-batch-overlap \
--mem-fraction-static 0.85 \
--chunked-prefill-size 524288 \
--max-running-requests 8192 \
--max-total-tokens 131072 \
--context-length 8192 \
--ep-num-redundant-experts 16 \
--ep-dispatch-algorithm dynamic \
--eplb-algorithm deepseek > /prefill.log 2>&1 &

SGLANG_NUM_RESERVED_DECODE_TOKENS=512 \
MC_TE_METRIC=true \
SGLANG_TBO_DEBUG=1 \
GLOO_SOCKET_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
python3 -m sglang.launch_server \
--model-path /upfs/models/deepseek-ai/DeepSeek-R1-0528 \
--disaggregation-ib-device mlx5_1 \
--disaggregation-mode decode \
--dist-init-addr 192.168.191.130:5000 \
--nnodes 6 \
--node-rank 2 \
--tp-size 48 \
--dp-size 48 \
--enable-dp-attention \
--page-size 1 \
--host 0.0.0.0 \
--trust-remote-code \
--moe-dense-tp-size 1 \
--enable-dp-lm-head \
--disable-radix-cache \
--watchdog-timeout 1000000 \
--enable-two-batch-overlap \
--mem-fraction-static 0.835 \
--max-running-requests 18432 \
--context-length 4500 \
--ep-num-redundant-experts 16 \
--cuda-graph-bs 256 > /decode.log 2>&1 &

my env
aiohappyeyeballs 2.6.1
aiohttp 3.12.13
aiosignal 1.3.2
airportsdata 20250622
annotated-types 0.7.0
anthropic 0.55.0
anyio 4.9.0
argcomplete 3.6.2
asttokens 3.0.0
async-timeout 5.0.1
attrs 25.3.0
black 25.1.0
blinker 1.4
blobfile 3.0.0
certifi 2025.6.15
cffi 1.17.1
cfgv 3.4.0
charset-normalizer 3.4.2
click 8.2.1
cloudpickle 3.1.1
compressed-tensors 0.10.2
cryptography 3.4.8
cuda-bindings 12.9.0
cuda-python 12.9.0
datamodel-code-generator 0.31.2
datasets 3.6.0
dbus-python 1.2.18
decorator 5.2.1
decord 0.6.0
deep_ep 1.1.0+bd429ff
devscripts 2.22.1ubuntu1
dill 0.3.8
diskcache 5.6.3
distlib 0.3.9
distro 1.7.0
einops 0.8.1
exceptiongroup 1.3.0
executing 2.2.0
fastapi 0.115.13
filelock 3.18.0
flashinfer-python 0.2.6.post1
frozenlist 1.7.0
fsspec 2025.3.0
genson 1.3.0
h11 0.16.0
hf_transfer 0.1.9
hf-xet 1.1.5
html5lib 1.1
httpcore 1.0.9
httplib2 0.20.2
httpx 0.28.1
huggingface-hub 0.33.1
icdiff 2.0.7
identify 2.6.12
idna 3.10
importlib_metadata 8.7.0
inflect 7.5.0
iniconfig 2.1.0
interegular 0.3.3
ipython 8.37.0
isort 6.0.1
jedi 0.19.2
jeepney 0.7.1
Jinja2 3.1.6
jiter 0.10.0
jsonschema 4.24.0
jsonschema-specifications 2025.4.1
keyring 23.5.0
lark 1.2.2
launchpadlib 1.10.16
lazr.restfulclient 0.14.4
lazr.uri 1.0.6
litellm 1.73.1
llguidance 0.7.30
lxml 5.4.0
MarkupSafe 3.0.2
matplotlib-inline 0.1.7
modelscope 1.27.1
mooncake-transfer-engine 0.3.4.post2
more-itertools 8.10.0
mpmath 1.3.0
msgspec 0.19.0
multidict 6.5.1
multiprocess 0.70.16
mypy_extensions 1.1.0
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.4
nodeenv 1.9.1
numpy 2.2.6
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 12.575.51
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvtx-cu12 12.6.77
oauthlib 3.2.0
openai 1.91.0
orjson 3.10.18
outlines 0.1.11
outlines_core 0.1.26
packaging 25.0
pandas 2.3.0
parso 0.8.4
partial-json-parser 0.2.1.1.post6
pathspec 0.12.1
pexpect 4.9.0
pillow 11.2.1
pip 25.1.1
platformdirs 4.3.8
pluggy 1.6.0
pre_commit 4.2.0
prometheus_client 0.22.1
prompt_toolkit 3.0.51
propcache 0.3.2
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 20.0.0
pycountry 24.6.1
pycparser 2.22
pycryptodomex 3.23.0
pydantic 2.11.7
pydantic_core 2.33.2
Pygments 2.19.2
PyGObject 3.42.1
PyJWT 2.3.0
pynvml 12.0.0
pyparsing 2.4.7
pytest 8.4.1
python-apt 2.4.0+ubuntu4
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.0.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.4
rpds-py 0.25.1
safetensors 0.5.3
scikit_build_core 0.11.4
scipy 1.15.3
SecretStorage 3.3.1
sentencepiece 0.2.0
setproctitle 1.3.6
setuptools 80.9.0
sgl-kernel 0.1.9
sglang 0.4.8.post1 /sgl-workspace/sglang/python
six 1.17.0
sniffio 1.3.1
soundfile 0.13.1
stack-data 0.6.3
starlette 0.46.2
sympy 1.14.0
tiktoken 0.9.0
tokenizers 0.21.2
tomli 2.2.1
torch 2.7.1+cu126
torch_memory_saver 0.0.8
torchao 0.9.0+cu126
torchaudio 2.7.1+cu126
torchvision 0.22.1+cu126
tqdm 4.67.1
traitlets 5.14.3
transformers 4.52.3
triton 3.3.1
typeguard 4.4.4
typing_extensions 4.14.0
typing-inspection 0.4.1
tzdata 2025.2
urllib3 2.5.0
uv 0.7.15
uvicorn 0.34.3
uvloop 0.21.0
virtualenv 20.31.2
wadllib 1.3.6
wcwidth 0.2.13
webencodings 0.5.1
wheel 0.45.1
xgrammar 0.1.19
xxhash 3.5.0
yarl 1.20.1
zipp 3.23.0

CAN SOMEONE HELP ME?
THANK U SO MUCH

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

Originally created by @mugglew on GitHub (Jul 7, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/597 ### Bug Report When i starts the first prefill node it reports error log below: `[2025-07-06 23:33:30 DP6 TP6] Try DeepGEMM JIT Compiling for <gemm_fp8_fp8_bf16_nt> N=7168, K=18432, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run python3 -m sglang.compile_deep_gemm`. 100%|██████████| 16/16 [00:09<00:00, 1.77it/s] [2025-07-06 23:33:41 DP0 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 127, in forward_thread_func self.forward_thread_func_() File "/usr/local/lib/python3.10/dist-packages/torch/utils/contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 162, in forward_thread_func self.worker.forward_batch_generation( File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 212, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1231, in forward output = self._forward_raw( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1260, in _forward_raw ret = self.forward_extend( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1199, in forward_extend return self.model.forward( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1962, in forward hidden_states = self.model(input_ids, positions, forward_batch, input_embeds) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1860, in forward hidden_states, residual = model_forward_maybe_tbo( File "/sgl-workspace/sglang/python/sglang/srt/two_batch_overlap.py", line 462, in model_forward_maybe_tbo return _model_forward_tbo( File "/sgl-workspace/sglang/python/sglang/srt/two_batch_overlap.py", line 488, in _model_forward_tbo outputs_arr = execute_overlapped_operations( File "/sgl-workspace/sglang/python/sglang/srt/operations.py", line 44, in execute_overlapped_operations executor_a.next() File "/sgl-workspace/sglang/python/sglang/srt/operations.py", line 84, in next self._stage_output = op.fn( File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1741, in op_comm_prepare_mlp self.layer_communicator.prepare_mlp( File "/sgl-workspace/sglang/python/sglang/srt/layers/communicator.py", line 205, in prepare_mlp return self._communicate_with_all_reduce_and_layer_norm_fn( File "/sgl-workspace/sglang/python/sglang/srt/layers/communicator.py", line 395, in _gather_hidden_states_and_residual dp_gather_partial(hidden_states, local_hidden_states, forward_batch) File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 269, in dp_gather_partial _dp_gather(global_tokens, local_tokens, forward_batch, is_partial=True) File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 232, in _dp_gather local_start_pos, local_num_tokens = get_dp_local_info(forward_batch) File "/sgl-workspace/sglang/python/sglang/srt/layers/dp_attention.py", line 172, in get_dp_local_info cumtokens = torch.cumsum(forward_batch.global_num_tokens_gpu, dim=0) TypeError: cumsum() received an invalid combination of arguments - got (NoneType, dim=int), but expected one of: (Tensor input, int dim, *, torch.dtype dtype = None, Tensor out = None) (Tensor input, name dim, *, torch.dtype dtype = None, Tensor out = None) 100%|██████████| 16/16 [00:08<00:00, 1.81it/s] [2025-07-06 23:33:41] Child process unexpectedly failed with exitcode=131. pid=9808` my command ```shell MC_TE_METRIC=true \ GLOO_SOCKET_IFNAME=eth0 \ NCCL_SOCKET_IFNAME=eth0 \ SGLANG_TBO_DEBUG=1 \ python3 -m sglang.launch_server \ --model-path /upfs/models/deepseek-ai/DeepSeek-R1-0528 \ --disaggregation-ib-device mlx5_1 \ --disaggregation-mode prefill \ --dist-init-addr 192.168.159.48:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --dp-size 16 \ --enable-dp-attention \ --page-size 1 \ --host 0.0.0.0 \ --trust-remote-code \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ --disable-radix-cache \ --watchdog-timeout 1000000 \ --enable-two-batch-overlap \ --mem-fraction-static 0.85 \ --chunked-prefill-size 524288 \ --max-running-requests 8192 \ --max-total-tokens 131072 \ --context-length 8192 \ --ep-num-redundant-experts 16 \ --ep-dispatch-algorithm dynamic \ --eplb-algorithm deepseek > /prefill.log 2>&1 & SGLANG_NUM_RESERVED_DECODE_TOKENS=512 \ MC_TE_METRIC=true \ SGLANG_TBO_DEBUG=1 \ GLOO_SOCKET_IFNAME=eth0 \ NCCL_SOCKET_IFNAME=eth0 \ python3 -m sglang.launch_server \ --model-path /upfs/models/deepseek-ai/DeepSeek-R1-0528 \ --disaggregation-ib-device mlx5_1 \ --disaggregation-mode decode \ --dist-init-addr 192.168.191.130:5000 \ --nnodes 6 \ --node-rank 2 \ --tp-size 48 \ --dp-size 48 \ --enable-dp-attention \ --page-size 1 \ --host 0.0.0.0 \ --trust-remote-code \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ --disable-radix-cache \ --watchdog-timeout 1000000 \ --enable-two-batch-overlap \ --mem-fraction-static 0.835 \ --max-running-requests 18432 \ --context-length 4500 \ --ep-num-redundant-experts 16 \ --cuda-graph-bs 256 > /decode.log 2>&1 & ``` my env aiohappyeyeballs 2.6.1 aiohttp 3.12.13 aiosignal 1.3.2 airportsdata 20250622 annotated-types 0.7.0 anthropic 0.55.0 anyio 4.9.0 argcomplete 3.6.2 asttokens 3.0.0 async-timeout 5.0.1 attrs 25.3.0 black 25.1.0 blinker 1.4 blobfile 3.0.0 certifi 2025.6.15 cffi 1.17.1 cfgv 3.4.0 charset-normalizer 3.4.2 click 8.2.1 cloudpickle 3.1.1 compressed-tensors 0.10.2 cryptography 3.4.8 cuda-bindings 12.9.0 cuda-python 12.9.0 datamodel-code-generator 0.31.2 datasets 3.6.0 dbus-python 1.2.18 decorator 5.2.1 decord 0.6.0 deep_ep 1.1.0+bd429ff devscripts 2.22.1ubuntu1 dill 0.3.8 diskcache 5.6.3 distlib 0.3.9 distro 1.7.0 einops 0.8.1 exceptiongroup 1.3.0 executing 2.2.0 fastapi 0.115.13 filelock 3.18.0 flashinfer-python 0.2.6.post1 frozenlist 1.7.0 fsspec 2025.3.0 genson 1.3.0 h11 0.16.0 hf_transfer 0.1.9 hf-xet 1.1.5 html5lib 1.1 httpcore 1.0.9 httplib2 0.20.2 httpx 0.28.1 huggingface-hub 0.33.1 icdiff 2.0.7 identify 2.6.12 idna 3.10 importlib_metadata 8.7.0 inflect 7.5.0 iniconfig 2.1.0 interegular 0.3.3 ipython 8.37.0 isort 6.0.1 jedi 0.19.2 jeepney 0.7.1 Jinja2 3.1.6 jiter 0.10.0 jsonschema 4.24.0 jsonschema-specifications 2025.4.1 keyring 23.5.0 lark 1.2.2 launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6 litellm 1.73.1 llguidance 0.7.30 lxml 5.4.0 MarkupSafe 3.0.2 matplotlib-inline 0.1.7 modelscope 1.27.1 mooncake-transfer-engine 0.3.4.post2 more-itertools 8.10.0 mpmath 1.3.0 msgspec 0.19.0 multidict 6.5.1 multiprocess 0.70.16 mypy_extensions 1.1.0 nest-asyncio 1.6.0 networkx 3.4.2 ninja 1.11.1.4 nodeenv 1.9.1 numpy 2.2.6 nvidia-cublas-cu12 12.6.4.1 nvidia-cuda-cupti-cu12 12.6.80 nvidia-cuda-nvrtc-cu12 12.6.77 nvidia-cuda-runtime-cu12 12.6.77 nvidia-cudnn-cu12 9.5.1.17 nvidia-cufft-cu12 11.3.0.4 nvidia-cufile-cu12 1.11.1.6 nvidia-curand-cu12 10.3.7.77 nvidia-cusolver-cu12 11.7.1.2 nvidia-cusparse-cu12 12.5.4.2 nvidia-cusparselt-cu12 0.6.3 nvidia-ml-py 12.575.51 nvidia-nccl-cu12 2.26.2 nvidia-nvjitlink-cu12 12.6.85 nvidia-nvtx-cu12 12.6.77 oauthlib 3.2.0 openai 1.91.0 orjson 3.10.18 outlines 0.1.11 outlines_core 0.1.26 packaging 25.0 pandas 2.3.0 parso 0.8.4 partial-json-parser 0.2.1.1.post6 pathspec 0.12.1 pexpect 4.9.0 pillow 11.2.1 pip 25.1.1 platformdirs 4.3.8 pluggy 1.6.0 pre_commit 4.2.0 prometheus_client 0.22.1 prompt_toolkit 3.0.51 propcache 0.3.2 psutil 7.0.0 ptyprocess 0.7.0 pure_eval 0.2.3 pyarrow 20.0.0 pycountry 24.6.1 pycparser 2.22 pycryptodomex 3.23.0 pydantic 2.11.7 pydantic_core 2.33.2 Pygments 2.19.2 PyGObject 3.42.1 PyJWT 2.3.0 pynvml 12.0.0 pyparsing 2.4.7 pytest 8.4.1 python-apt 2.4.0+ubuntu4 python-dateutil 2.9.0.post0 python-dotenv 1.1.1 python-multipart 0.0.20 pytz 2025.2 PyYAML 6.0.2 pyzmq 27.0.0 referencing 0.36.2 regex 2024.11.6 requests 2.32.4 rpds-py 0.25.1 safetensors 0.5.3 scikit_build_core 0.11.4 scipy 1.15.3 SecretStorage 3.3.1 sentencepiece 0.2.0 setproctitle 1.3.6 setuptools 80.9.0 sgl-kernel 0.1.9 sglang 0.4.8.post1 /sgl-workspace/sglang/python six 1.17.0 sniffio 1.3.1 soundfile 0.13.1 stack-data 0.6.3 starlette 0.46.2 sympy 1.14.0 tiktoken 0.9.0 tokenizers 0.21.2 tomli 2.2.1 torch 2.7.1+cu126 torch_memory_saver 0.0.8 torchao 0.9.0+cu126 torchaudio 2.7.1+cu126 torchvision 0.22.1+cu126 tqdm 4.67.1 traitlets 5.14.3 transformers 4.52.3 triton 3.3.1 typeguard 4.4.4 typing_extensions 4.14.0 typing-inspection 0.4.1 tzdata 2025.2 urllib3 2.5.0 uv 0.7.15 uvicorn 0.34.3 uvloop 0.21.0 virtualenv 20.31.2 wadllib 1.3.6 wcwidth 0.2.13 webencodings 0.5.1 wheel 0.45.1 xgrammar 0.1.19 xxhash 3.5.0 yarl 1.20.1 zipp 3.23.0 CAN SOMEONE HELP ME? THANK U SO MUCH ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi 2026-10-29 21:10:48 +00:00

• closed this issue
• added the
  stale
  auto-closed
  bug
  labels

bsakaguchi commented 2026-10-29 21:10:49 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 30, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4839876618 --> @github-actions[bot] commented on GitHub (Jun 30, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:10:49 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 7, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4900236687 --> @github-actions[bot] commented on GitHub (Jul 7, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:26:59 +00:00

[PR #199] [MERGED] [CI] Update etcd-cpp CMake policy version in dependencies script #1389

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#199

No description provided.