mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #1426] [Bug]: The mooncake.store module must be imported before mooncake.transferengine; otherwise, a segmentation fault occurs. #459
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#459
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @chenhao-stick-to on GitHub (Jan 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1426
Bug Report
mooncake version：v0.3.8
system：almalinux9
As shown in the figure below, in a multithreaded environment, if we comment out the from mooncake.store import MooncakeDistributedStore inside init_store, and instead import mooncake.engine first followed by mooncake.store, a segmentation fault (as shown in the figure) occurs. However, if we import mooncake.store first, the issue disappears.
This problem arises when using SGLang: when its new rfork feature is enabled, it uses TransferEngine for fast model loading, and later initializes MooncakeStore (via HiCache). At that point, the aforementioned segmentation fault appears. The code snippet above is a minimal reproducible example of this issue.
The root cause seems to be related to some kind of race condition or initialization conflict between the two modules and coro_rpc.
I’ve already verified that this crash is 100% reproducible. There must be some underlying low-level issue here. If there’s already a known fix or workaround, please let me know immediately!!!
Before submitting...
@chenhao-stick-to commented on GitHub (Jan 23, 2026):
erros are follwed:
@00fish0 commented on GitHub (Jan 28, 2026):
Thanks for reporting this with a detailed reproducible example. I'll look into the root cause and see if I can provide a fix for this initialization conflict.
@00fish0 commented on GitHub (Jan 28, 2026):
Hi @chenhao-stick-to,
I tried to reproduce the segfault with your repro script (importing
mooncake.enginebeforemooncake.storein a multithreaded init), but I still can’t reproduce it on my side. I tested both mooncake v0.3.8 and the current commit (8383b93), and the behavior/logs look the same: 8/8 threads initialize successfully, no crash.Environment:
8383b93Log excerpt (same on both versions):
If there’s any specific config or build flag to match your environment more closely, please let me know and I can try again.
@chenhao-stick-to commented on GitHub (Jan 28, 2026):
`import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import List, Optional, Tuple
import os
import time
import sys
local_hostname = "10.148.0.152"
metadata_server = "P2PHANDSHAKE"
master_server_address = "10.45.180.240:50051"
protocol = os.getenv("MOONCAKE_PROTOCOL", "tcp")
device_name = ""
global_segment_size = 449755813
DEFAULT_LOCAL_BUFFER_SIZE = 1610241024
def init_transfer(num_threads: int = 8):
from mooncake.engine import TransferEngine
remote_instance_transfer_engine = TransferEngine()
remote_instance_transfer_engine.initialize(
local_hostname, "P2PHANDSHAKE", protocol, device_name
)
print("begin mooncake-transferengine")
time.sleep(10)
def init_mooncakestore():
print("begine mooncake-store")
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
print("tmp")
ret_code = store.setup( local_hostname, metadata_server, global_segment_size,DEFAULT_LOCAL_BUFFER_SIZE, protocol, device_name, master_server_address)
time.sleep(30)
def init_mooncake_multi_thread(num_threads: int = 8) -> Tuple[bool, List, List]:
stores = [None] * num_threads
errors = [None] * num_threads
def init_store(thread_id: int):
try：
init_transfer()
init_mooncakestore()
if ret_code:
return (thread_id, None, f"Setup failed: {ret_code}")
return (thread_id, store, None)
except Exception as e:
return (thread_id, None, str(e))
init_mooncake_multi_thread(8)`
I used this script to reproduce the issue。could you try again？
@00fish0 commented on GitHub (Jan 28, 2026):
Thanks for providing the updated reproduction script.
I have tested this new script using mooncake v0.3.8 in my environment, but unfortunately, I am still unable to reproduce the segmentation fault. The issue might be tied to specific environmental factors.
@chenhao-stick-to commented on GitHub (Jan 28, 2026):
ok，thank you for your work；may is the environment problem.lastly,i used the v0.5.7 version of yalantinglibs to compile,worked it out
@github-actions[bot] commented on GitHub (Jun 9, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Jun 16, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.