mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #2201] [Bug]: 华为昇腾910c上使用ModeB dummy client方案卸载SSD，decode节点启动plog报大量的FuncErrorReason:rtMem cpy execution failed, reason=the current capture mode does not support this operation #609
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#609
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xuyucheng220 on GitHub (May 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2201
Bug Report
decode plog日志：
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.186.911 [api_impl.cc:2725]73550 MemCopySync:The current capture
mode:threadCaptureMode=0, exchangeCaptureMode=0, contextCaptureMode=0
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.186.917 [api_error.cc:1612]73550 MemCopySync:Memory copy sync f
ailed, cnt=456, kind=1.
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.186.924 [api_c.cc:1081]73550 rtMemcpy:ErrCode=107030, desc=[the
current capture mode does not support this operation], InnerCode=0x703001c
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.186.928 [error_message_manage.cc:65]73550 FuncErrorReason:rtMem
cpy execution failed, reason=the current capture mode does not support this operation
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] ASCENDCL(73550,):2026-05-24-09:42:40.187.239 [memory.cpp:608]73550 aclrtMemcpyImpl:synchronized mem
cpy failed, kind = 1, runtime result = 107030
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.190.214 [api_impl.cc:2725]73550 MemCopySync:The current capture
mode:threadCaptureMode=0, exchangeCaptureMode=0, contextCaptureMode=0
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.190.218 [api_error.cc:1612]73550 MemCopySync:Memory copy sync f
ailed, cnt=72, kind=1.
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.190.224 [api_c.cc:1081]73550 rtMemcpy:ErrCode=107030, desc=[the
current capture mode does not support this operation], InnerCode=0x703001c
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.190.226 [error_message_manage.cc:65]73550 FuncErrorReason:rtMem
cpy execution failed, reason=the current capture mode does not support this operation
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] ASCENDCL(73550,):2026-05-24-09:42:40.190.381 [memory.cpp:608]73550 aclrtMemcpyImpl:synchronized mem
cpy failed, kind = 1, runtime result = 107030
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.192.587 [api_impl.cc:2725]73550 MemCopySync:The current capture
mode:threadCaptureMode=0, exchangeCaptureMode=0, contextCaptureMode=0
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.192.592 [api_error.cc:1612]73550 MemCopySync:Memory copy sync f
ailed, cnt=456, kind=1.
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.192.597 [api_c.cc:1081]73550 rtMemcpy:ErrCode=107030, desc=[the
current capture mode does not support this operation], InnerCode=0x703001c
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] RUNTIME(73550,):2026-05-24-09:42:40.192.600 [error_message_manage.cc:65]73550 FuncErrorReason:rtMem
cpy execution failed, reason=the current capture mode does not support this operation
/root/ascend/log/debug/plog/plog-73550_20260524094240186.log:[ERROR] ASCENDCL(73550,):2026-05-24-09:42:40.192.687 [memory.cpp:608]73550 aclrtMemcpyImpl:synchronized mem
--More--
decode启动命令：
more run_dp_template.sh
nic_name="enp23s0f3" # change to your own nic name
local_ip="172.16.0.49" # change to your own ip
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/data2/mooncake_offload
export MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=10737418240
export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
export MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY=lru
#export MOONCAKE_OFFLOAD_SCANMETA_ITERATOR_KEYS_LIMIT=20000
#export MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES=2684354560
#export MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT=5000
#1T
#export MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE=1099511627776
#10G
export MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE=10737418240
#1T
#export MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES=1024000000000
#10G
export MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES=10737418240
#export HCCL_HOST_SOCKET_PORT_RANGE="auto"
#export HCCL_NPU_SOCKET_PORT_RANGE="auto"
export HCCL_HOST_SOCKET_PORT_RANGE="61000-62000"
export HCCL_NPU_SOCKET_PORT_RANGE="61000-62000"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
#Mooncake
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=256
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
Timeout (in seconds) for automatically releasing the prefiller’s KV cache for a particular request.
export VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT=480
export TASK_QUEUE_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES=$1
#export VLLM_ASCEND_ENABLE_FUSED_MC2=1
#export HCCL_INTRA_PCIE_ENABLE=0
#export HCCL_INTRA_ROCE_ENABLE=0
export VLLM_ASCEND_ENABLE_MLAPO=1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
#mooncake kvcache dram
#export MOONCAKE_CONFIG_PATH="/mnt/sfs_turbo/glm5.1-pd/19.0rc1/762t/mooncake_ssd2.json"
export MOONCAKE_CONFIG_PATH="/mnt/sfs_turbo/glm5.1-pd/19.0rc1/762t/mooncake_ssd2_ha.json"
#export MOONCAKE_CONFIG_PATH="/mnt/sfs_turbo/glm5.1-pd/19.0rc1/762t/mooncake_ssd2_ha_modeA.json"
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH
unset ASCEND_ENABLE_USE_FABRIC_MEM
export ASCEND_ENABLE_USE_FABRIC_MEM=1
unset ASCEND_BUFFER_POOL
#export ASCEND_BUFFER_POOL=4:8
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=100000
export ASCEND_TRANSFER_TIMEOUT=100000
model_path="/data1/tmpfs_model/GLM-5.1-W8A8/"
#model_path="/mnt/sfs_turbo/GLM-5.1-W8A8/"
vllm serve $model_path
--host 0.0.0.0
--port $2
--data-parallel-size $3
--data-parallel-address $5
--data-parallel-rpc-port $6
--tensor-parallel-size $7
--speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}'
--profiler-config
'{"profiler": "torch",
"torch_profiler_dir": "./vllm_profile",
"torch_profiler_with_stack": false}'
--seed 1024
--served-model-name glm-5
--max-model-len 69000
--enable-prefix-caching
--max-num-batched-tokens 16
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY", "cudagraph_capture_sizes":[4, 8, 12, 16,20,24,28, 32]}'
--additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "ascend_compilation_config": {"enable_np
ugraph_ex": true}}'
--trust-remote-code
--max-num-seqs 8
--gpu-memory-utilization 0.92
--async-scheduling
--quantization ascend
--enable-auto-tool-choice
--tool-call-parser glm47
--reasoning-parser glm45
--kv-transfer-config
'{
"kv_connector": "MultiConnector",
"kv_role": "kv_consumer",
"kv_connector_extra_config": {
"connectors": [
{
"kv_connector": "MooncakeConnectorV1",
"kv_role": "kv_consumer",
"kv_port": "30100",
"kv_connector_extra_config": {
"prefill": {
"dp_size": 1,
"tp_size": 16
},
"decode": {
"dp_size": 1,
"tp_size": 16
}
}
},
{
"kv_connector": "AscendStoreConnector",
"kv_role": "kv_consumer",
"kv_connector_extra_config": {
"lookup_rpc_port":"0",
"backend": "mooncake"
}
}
]
}
}' > /data1/vllm_log/glm5.log 2>&1 &
Before submitting...
@ascend-direct-dev commented on GitHub (May 25, 2026):
This appears to be a known model execution error. It should not affect normal execution and is unrelated to Mooncake. Please refer to the vLLM Ascend community for related issues.