kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #3314] [Bug]: mooncake-transfer-engine-non-cuda requires libcudart.so.12. #839

New issue
Closed
opened 2026-10-29 21:20:04 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 21:20:04 +00:00
Owner
Copy link

Originally created by @elpis-furiosa on GitHub (Aug 6, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3314

Bug Report

I am currently using mooncake-transfer-engine-non-cuda==0.3.12.post1 in order to launch mooncake_master. When I try to launch mooncake_master, the following error occurs:

❯ mooncake_master
/home/sukchul/test-venv/lib/python3.10/site-packages/mooncake/mooncake_master: error while loading shared libraries: libcudart.so.12: cannot open shared object file: No such file or directory

It seems that the store.so bundled inside the package requires libcudart.so.12. Is this intended?

❯ find ./test-venv -name "store.so"
./test-venv/lib/python3.10/site-packages/mooncake/store.so
❯ readelf -dW ./test-venv/lib/python3.10/site-packages/mooncake/store.so

Dynamic section at offset 0x1037000 contains 45 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN:$ORIGIN/../mooncake_transfer_engine_non_cuda.libs]
 0x0000000000000001 (NEEDED)             Shared library: [libetcd_wrapper.so]
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libjsoncpp-4b349785.so.1.8.4]
 0x0000000000000001 (NEEDED)             Shared library: [libibverbs.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libnuma.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libglog-6ed04f2c.so.0.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgflags-8aee0f6c.so.2.1.2]
 0x0000000000000001 (NEEDED)             Shared library: [libasio.so]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libxxhash-3b878866.so.0.8.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzstd-abac8a45.so.1.5.5]
 0x0000000000000001 (NEEDED)             Shared library: [liburing-2907b6bc.so.1.0.7]
 0x0000000000000001 (NEEDED)             Shared library: [libyaml-cpp-2ec4b324.so.0.7.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcudart.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000c (INIT)               0x243000
 0x000000000000000d (FINI)               0xc2f6f4
 0x0000000000000019 (INIT_ARRAY)         0xe5d3c0
 0x000000000000001b (INIT_ARRAYSZ)       504 (bytes)
 0x000000000000001a (FINI_ARRAY)         0xe5d5b8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0xe86028
 0x0000000000000005 (STRTAB)             0x1045320
 0x0000000000000006 (SYMTAB)             0x134c8
 0x000000000000000a (STRSZ)              1749613 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0xe69fe8
 0x0000000000000002 (PLTRELSZ)           121008 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x224d80
 0x0000000000000007 (RELA)               0x1ffdf0
 0x0000000000000008 (RELASZ)             151440 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x1ffa30
 0x000000006fffffff (VERNEEDNUM)         11
 0x000000006ffffff0 (VERSYM)             0x1fa9e2
 0x000000006ffffff9 (RELACOUNT)          1724
 0x0000000000000000 (NULL)               0x0

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @elpis-furiosa on GitHub (Aug 6, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3314 ### Bug Report I am currently using `mooncake-transfer-engine-non-cuda==0.3.12.post1` in order to launch `mooncake_master`. When I try to launch `mooncake_master`, the following error occurs: ``` ❯ mooncake_master /home/sukchul/test-venv/lib/python3.10/site-packages/mooncake/mooncake_master: error while loading shared libraries: libcudart.so.12: cannot open shared object file: No such file or directory ``` It seems that the `store.so` bundled inside the package requires `libcudart.so.12`. Is this intended? ``` ❯ find ./test-venv -name "store.so" ./test-venv/lib/python3.10/site-packages/mooncake/store.so ❯ readelf -dW ./test-venv/lib/python3.10/site-packages/mooncake/store.so Dynamic section at offset 0x1037000 contains 45 entries: Tag Type Name/Value 0x000000000000000f (RPATH) Library rpath: [$ORIGIN:$ORIGIN/../mooncake_transfer_engine_non_cuda.libs] 0x0000000000000001 (NEEDED) Shared library: [libetcd_wrapper.so] 0x0000000000000001 (NEEDED) Shared library: [libcurl.so.4] 0x0000000000000001 (NEEDED) Shared library: [libjsoncpp-4b349785.so.1.8.4] 0x0000000000000001 (NEEDED) Shared library: [libibverbs.so.1] 0x0000000000000001 (NEEDED) Shared library: [libnuma.so.1] 0x0000000000000001 (NEEDED) Shared library: [libglog-6ed04f2c.so.0.0.0] 0x0000000000000001 (NEEDED) Shared library: [libgflags-8aee0f6c.so.2.1.2] 0x0000000000000001 (NEEDED) Shared library: [libasio.so] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libxxhash-3b878866.so.0.8.2] 0x0000000000000001 (NEEDED) Shared library: [libzstd-abac8a45.so.1.5.5] 0x0000000000000001 (NEEDED) Shared library: [liburing-2907b6bc.so.1.0.7] 0x0000000000000001 (NEEDED) Shared library: [libyaml-cpp-2ec4b324.so.0.7.0] 0x0000000000000001 (NEEDED) Shared library: [libcudart.so.12] 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [librt.so.1] 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] 0x000000000000000c (INIT) 0x243000 0x000000000000000d (FINI) 0xc2f6f4 0x0000000000000019 (INIT_ARRAY) 0xe5d3c0 0x000000000000001b (INIT_ARRAYSZ) 504 (bytes) 0x000000000000001a (FINI_ARRAY) 0xe5d5b8 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0xe86028 0x0000000000000005 (STRTAB) 0x1045320 0x0000000000000006 (SYMTAB) 0x134c8 0x000000000000000a (STRSZ) 1749613 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0xe69fe8 0x0000000000000002 (PLTRELSZ) 121008 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x224d80 0x0000000000000007 (RELA) 0x1ffdf0 0x0000000000000008 (RELASZ) 151440 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x1ffa30 0x000000006fffffff (VERNEEDNUM) 11 0x000000006ffffff0 (VERSYM) 0x1fa9e2 0x000000006ffffff9 (RELACOUNT) 1724 0x0000000000000000 (NULL) 0x0 ``` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:20:04 +00:00
  • closed this issue
  • added the
    bug
         label
bsakaguchi commented 2026-10-29 21:20:05 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Aug 6, 2026):

Thanks for opening this issue, @elpis-furiosa!

Field Value
Issue #3314
GitHub user ID 188016236
Reporter @elpis-furiosa

A maintainer will triage this when possible. To help us respond faster, please include:

  • Mooncake version or commit SHA
  • Environment (OS, CUDA/driver, RDMA stack if relevant)
  • Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:5204058271 --> @github-actions[bot] commented on GitHub (Aug 6, 2026): Thanks for opening this issue, @elpis-furiosa! | Field | Value | |-------|-------| | **Issue** | #3314 | | **GitHub user ID** | `188016236` | | **Reporter** | @elpis-furiosa | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.
bsakaguchi commented 2026-10-29 21:20:06 +00:00
Author
Owner
Copy link

@elpis-furiosa commented on GitHub (Aug 6, 2026):

I just found out that this issue is resolved here. I'd like to know whether a 0.3.12.post2 release is planned, or whether the plan is to go straight to 0.3.13.

<!-- gh-comment-id:5204158192 --> @elpis-furiosa commented on GitHub (Aug 6, 2026): I just found out that this issue is resolved [here](https://github.com/kvcache-ai/Mooncake/commit/d7acfd608153afe757d8919c9c76406426437f5a). I'd like to know whether a 0.3.12.post2 release is planned, or whether the plan is to go straight to 0.3.13.
bsakaguchi commented 2026-10-29 21:20:06 +00:00
Author
Owner
Copy link

@ykwd commented on GitHub (Aug 7, 2026):

Sorry for the inconvenience. The wheel packages that have already been published to PyPI cannot be modified, so we’re planning to fix this issue in the next release.

We ran into quite a few issues during the previous release, and in response, we’ve put together a revised release process to help avoid similar problems going forward: https://github.com/kvcache-ai/Mooncake/issues/3130

<!-- gh-comment-id:5213612782 --> @ykwd commented on GitHub (Aug 7, 2026): Sorry for the inconvenience. The wheel packages that have already been published to PyPI cannot be modified, so we’re planning to fix this issue in the next release. We ran into quite a few issues during the previous release, and in response, we’ve put together a revised release process to help avoid similar problems going forward: https://github.com/kvcache-ai/Mooncake/issues/3130
bsakaguchi commented 2026-10-29 21:20:06 +00:00
Author
Owner
Copy link

@elpis-furiosa commented on GitHub (Aug 7, 2026):

Thank you for letting me know! I'll wait for the next release to happen.

<!-- gh-comment-id:5215022874 --> @elpis-furiosa commented on GitHub (Aug 7, 2026): Thank you for letting me know! I'll wait for the next release to happen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#839
No description provided.