kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #1107] [Bug]: Mooncake RDMA error on Ascend 910B #368

New issue
Closed
opened 2026-10-29 21:13:28 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-29 21:13:28 +00:00
Owner
Copy link

Originally created by @heijian123 on GitHub (Nov 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1107

Bug Report

We encountered an error when starting Mooncake 0.3.7 using RDMA on Ascend 910B. The configuration is as follows:

chunk_size: 512
local_device: "cpu"
remote_url: "mooncakestore://xxx:50003"
remote_serde: "naive"
local_cpu: False
max_local_cpu_size: 25
external_lookup_client: "mooncakestore://xxx:50003"

extra_config:
  use_exists_sync: true
  save_chunk_meta: False
  local_hostname: "xxx"
  metadata_server: "redis://xxx:6579"
  protocol: "rdma"
  device_name: ""
  master_server_address: "xxx:50003"
  global_segment_size: 161061273600
  local_buffer_size: 12884901888
  transfer_timeout: 5

Error message：

I1125 03:04:35.395231 25810 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 10.44.187.209:15669
I1125 03:04:35.395501 25810 transfer_engine.cpp:185] Auto-discovering topology...
I1125 03:04:35.400758 25810 transfer_engine.cpp:200] Topology discovery complete. Found 2 HCAs.
I1125 03:04:35.409325 25811 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ 
I1125 03:04:35.410225 25811 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1
I1125 03:04:35.449532 25812 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ 
I1125 03:04:35.450400 25812 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1
I1125 03:04:35.450781 25811 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ 
I1125 03:04:35.451742 25811 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83
I1125 03:04:35.452214 25811 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma
I1125 03:04:35.461421 25813 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ 
I1125 03:04:35.462273 25813 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83
I1125 03:04:35.462688 25813 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma
I1125 03:04:35.466563 25810 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ 
I1125 03:04:35.467550 25810 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1
I1125 03:04:35.494633 25810 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ 
I1125 03:04:35.495455 25810 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83
I1125 03:04:35.495865 25810 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma
I1125 03:04:35.497954 25812 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ 
I1125 03:04:35.498875 25812 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83
I1125 03:04:35.499330 25812 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma
I1125 03:04:37.593384 25811 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600
I1125 03:04:38.256182 25810 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600
I1125 03:04:38.388006 25813 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600
I1125 03:04:38.620812 25812 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600
(VllmWorker rank=3 pid=25813) [2025-11-25 03:05:05,130] LMCache INFO: Mooncake store setup completed successfully (mooncakestore_connector.py:161:lmcache.v1.storage_backend.connector.mooncakestore_connector)
E1125 03:05:05.279587 27052 rdma_context.cpp:243] Failed to register memory 0x12c180000000: Invalid argument [22]
E1125 03:05:05.427743 27057 rdma_context.cpp:243] Failed to register memory 0x12c374000000: Invalid argument [22]

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @heijian123 on GitHub (Nov 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1107 ### Bug Report We encountered an error when starting Mooncake 0.3.7 using RDMA on Ascend 910B. The configuration is as follows: ``` chunk_size: 512 local_device: "cpu" remote_url: "mooncakestore://xxx:50003" remote_serde: "naive" local_cpu: False max_local_cpu_size: 25 external_lookup_client: "mooncakestore://xxx:50003" extra_config: use_exists_sync: true save_chunk_meta: False local_hostname: "xxx" metadata_server: "redis://xxx:6579" protocol: "rdma" device_name: "" master_server_address: "xxx:50003" global_segment_size: 161061273600 local_buffer_size: 12884901888 transfer_timeout: 5 ``` Error message： ``` I1125 03:04:35.395231 25810 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 10.44.187.209:15669 I1125 03:04:35.395501 25810 transfer_engine.cpp:185] Auto-discovering topology... I1125 03:04:35.400758 25810 transfer_engine.cpp:200] Topology discovery complete. Found 2 HCAs. I1125 03:04:35.409325 25811 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ I1125 03:04:35.410225 25811 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1 I1125 03:04:35.449532 25812 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ I1125 03:04:35.450400 25812 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1 I1125 03:04:35.450781 25811 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ I1125 03:04:35.451742 25811 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83 I1125 03:04:35.452214 25811 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma I1125 03:04:35.461421 25813 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ I1125 03:04:35.462273 25813 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83 I1125 03:04:35.462688 25813 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma I1125 03:04:35.466563 25810 rdma_context.cpp:533] Find best gid index: 3 on mlx5_0/ I1125 03:04:35.467550 25810 rdma_context.cpp:126] RDMA device: mlx5_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:0a:2c:bb:d1 I1125 03:04:35.494633 25810 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ I1125 03:04:35.495455 25810 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83 I1125 03:04:35.495865 25810 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma I1125 03:04:35.497954 25812 rdma_context.cpp:533] Find best gid index: 3 on mlx5_1/ I1125 03:04:35.498875 25812 rdma_context.cpp:126] RDMA device: mlx5_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:08:65:1a:83 I1125 03:04:35.499330 25812 client.cpp:264] Transfer engine auto discovery is enabled for protocol: rdma I1125 03:04:37.593384 25811 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600 I1125 03:04:38.256182 25810 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600 I1125 03:04:38.388006 25813 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600 I1125 03:04:38.620812 25812 pybind_client.cpp:218] Mounting segment: 161061273600 bytes, 161061273600 of 161061273600 (VllmWorker rank=3 pid=25813) [2025-11-25 03:05:05,130] LMCache INFO: Mooncake store setup completed successfully (mooncakestore_connector.py:161:lmcache.v1.storage_backend.connector.mooncakestore_connector) E1125 03:05:05.279587 27052 rdma_context.cpp:243] Failed to register memory 0x12c180000000: Invalid argument [22] E1125 03:05:05.427743 27057 rdma_context.cpp:243] Failed to register memory 0x12c374000000: Invalid argument [22] ``` ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:13:28 +00:00
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (Nov 25, 2025):

You can try to build Mooncake again with the --DUSE_ASCEND=ON option provided.

<!-- gh-comment-id:3574632827 --> @alogfans commented on GitHub (Nov 25, 2025): You can try to build Mooncake again with the `--DUSE_ASCEND=ON` option provided.
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@heijian123 commented on GitHub (Nov 26, 2025):

好的，我们试一下

<!-- gh-comment-id:3578494642 --> @heijian123 commented on GitHub (Nov 26, 2025): 好的，我们试一下
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@Keithwwa commented on GitHub (Nov 28, 2025):

what's the difference of DUSE_ASCEND and DUSE_ASCEND_DIRECT? @alogfans

<!-- gh-comment-id:3589191495 --> @Keithwwa commented on GitHub (Nov 28, 2025): what's the difference of DUSE_ASCEND and DUSE_ASCEND_DIRECT? @alogfans
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@tianlang-wq commented on GitHub (Mar 19, 2026):

我使用--DUSE_ASCEND_DIRECT=ON 可解决这个问题吗 @alogfans

<!-- gh-comment-id:4086989407 --> @tianlang-wq commented on GitHub (Mar 19, 2026): 我使用--DUSE_ASCEND_DIRECT=ON 可解决这个问题吗 @alogfans
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@tianlang-wq commented on GitHub (Mar 19, 2026):

@heijian123 请问问题解决了吗？

<!-- gh-comment-id:4087432241 --> @tianlang-wq commented on GitHub (Mar 19, 2026): @heijian123 请问问题解决了吗？
bsakaguchi commented 2026-10-29 21:13:29 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 18, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4738058490 --> @github-actions[bot] commented on GitHub (Jun 18, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:13:30 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 25, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4795955854 --> @github-actions[bot] commented on GitHub (Jun 25, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#368
No description provided.