[GH-ISSUE #1986] [Bug]: SIGSEGV in `MemcpyWorkerPool::workerThread` under TCP transport with concurrent put/get

bsakaguchi commented

2026-10-29 21:16:17 +00:00

Owner

Originally created by @Dogacel on GitHub (Apr 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1986

Originally assigned to: @fang-tech on GitHub.

Bug Report

Summary

On a TCP-only host, MC_STORE_MEMCPY auto-enables and creates a MemcpyWorkerPool. Under concurrent put/get from a multi-threaded Python client, a worker thread segfaults inside glibc __memcpy_avx512_unaligned_erms. Setting MC_STORE_MEMCPY=0 reliably avoids the crash.

Regression PR: https://github.com/kvcache-ai/Mooncake/pull/1936

Reproduction

Run a multi-process workload where one Mooncake client puts host buffers and another gets them concurrently over TCP.
After a few seconds (first batches arriving), a memcpy worker thread crashes.

I've used https://github.com/torchspec-project/TorchSpec's Qwen 8B training pipeline.

Crash signature

gdb on a core dump shows the crashing thread:

#0  __memcpy_avx512_unaligned_erms ()
    at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  mooncake::MemcpyWorkerPool::workerThread ()
    from .../site-packages/mooncake/store.so
#2  std::execute_native_thread_routine
#3  start_thread
#4  clone3

Workaround

export MC_STORE_MEMCPY=0

This bypasses MemcpyWorkerPool and the crash disappears completely. Switching transport to RDMA also avoids it (RDMA path doesn't use the pool).

Environment

mooncake-transfer-engine 0.3.10.post2 (pip)
Python 3.12.13 (miniconda)
Ubuntu, kernel 6.8.0-1043-aiext, glibc default, system libstdc++.so.6
Transport: TCP only (RDMA hardware available but not selected)
Workload: speculative-decoding training pipeline (TorchSpec). Inference workers call put_from, trainer workers call get_into host buffers concurrently.
Host buffer pool size: 8 × 1 GB; segment size 16 GB; local buffer 4 GB
MC_STORE_MEMCPY not set → log shows: auto-detected: TCP-only environment, memcpy enabled

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Dogacel on GitHub (Apr 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1986 Originally assigned to: @fang-tech on GitHub. ### Bug Report ## Summary On a TCP-only host, `MC_STORE_MEMCPY` auto-enables and creates a `MemcpyWorkerPool`. Under concurrent `put`/`get` from a multi-threaded Python client, a worker thread segfaults inside glibc `__memcpy_avx512_unaligned_erms`. Setting `MC_STORE_MEMCPY=0` reliably avoids the crash. Regression PR: https://github.com/kvcache-ai/Mooncake/pull/1936 ## Reproduction 1. Run a multi-process workload where one Mooncake client `put`s host buffers and another `get`s them concurrently over TCP. 2. After a few seconds (first batches arriving), a memcpy worker thread crashes. I've used https://github.com/torchspec-project/TorchSpec's Qwen 8B training pipeline. ## Crash signature `gdb` on a core dump shows the crashing thread: ``` #0 __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265 #1 mooncake::MemcpyWorkerPool::workerThread () from .../site-packages/mooncake/store.so #2 std::execute_native_thread_routine #3 start_thread #4 clone3 ``` ## Workaround ```bash export MC_STORE_MEMCPY=0 ``` This bypasses `MemcpyWorkerPool` and the crash disappears completely. Switching transport to RDMA also avoids it (RDMA path doesn't use the pool). --- ## Environment - `mooncake-transfer-engine` **0.3.10.post2** (pip) - Python 3.12.13 (miniconda) - Ubuntu, kernel `6.8.0-1043-aiext`, glibc default, system `libstdc++.so.6` - Transport: **TCP only** (RDMA hardware available but not selected) - Workload: speculative-decoding training pipeline (TorchSpec). Inference workers call `put_from`, trainer workers call `get_into` host buffers concurrently. - Host buffer pool size: 8 × 1 GB; segment size 16 GB; local buffer 4 GB - `MC_STORE_MEMCPY` not set → log shows: `auto-detected: TCP-only environment, memcpy enabled` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:16:17 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:16:18 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 27, 2026):

I'm working on this problem, plz assign it to me.

@fang-tech commented on GitHub (Apr 27, 2026): I'm working on this problem, plz assign it to me.

bsakaguchi commented

2026-10-29 21:16:18 +00:00

Author

Owner

@ykwd commented on GitHub (Apr 28, 2026):

I'm working on this problem, plz assign it to me.

Thanks a lot!

@ykwd commented on GitHub (Apr 28, 2026): > I'm working on this problem, plz assign it to me. Thanks a lot!

bsakaguchi commented

2026-10-29 21:16:19 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 29, 2026):

I'm working on this problem, plz assign it to me.

Please let me know if you have any fixes. Thanks.

@stmatengss commented on GitHub (Apr 29, 2026): > I'm working on this problem, plz assign it to me. Please let me know if you have any fixes. Thanks.

bsakaguchi commented

2026-10-29 21:16:19 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 29, 2026):

Root Cause Analysis

The branch point that leads to the memcpy path is selectStrategy() in mooncake-store/src/transfer_task.cpp. When MC_STORE_MEMCPY=1 (automatically set on TCP-only hosts) and isLocalTransfer() returns true, selectStrategy() returns TransferStrategy::LOCAL_MEMCPY.

The intended semantics of isLocalTransfer is to determine whether buffer_address_ can be directly dereferenced in the current process's virtual address space — i.e., whether memcpy(dest, buffer_address_, size) can be executed safely without going through the network. This condition holds only when the buffer was allocated by the current process itself. However, the current implementation only checks whether the buffer resides on the same host, not whether it belongs to the same process:

bool TransferSubmitter::isLocalTransfer(
    const AllocatedBuffer::Descriptor& handle) const {
    std::string local_ep = engine_.getLocalIpAndPort();
    std::string local_ip = extractIpAddress(local_ep);

    if (!local_ep.empty()) {
        std::string handle_ip = extractIpAddress(handle.transport_endpoint_);
        return !handle.transport_endpoint_.empty() && handle_ip == local_ip;
    }

    // Without a local IP we cannot prove locality; disable memcpy.
    return false;
}

Fix

Restore isLocalTransfer to compare the full IP:port instead of IP only.

Question

Historically the logic did perform a full IP:port comparison, but PR #1226 changed it to the current IP-only form. Was there a specific consideration behind that change? Understanding the intent would help confirm that this fix does not introduce any unintended regressions.
cc @stmatengss @ykwd

@fang-tech commented on GitHub (Apr 29, 2026): ## Root Cause Analysis The branch point that leads to the `memcpy` path is `selectStrategy()` in `mooncake-store/src/transfer_task.cpp`. When `MC_STORE_MEMCPY=1` (automatically set on TCP-only hosts) **and** `isLocalTransfer()` returns `true`, `selectStrategy()` returns `TransferStrategy::LOCAL_MEMCPY`. The intended semantics of `isLocalTransfer` is to determine whether `buffer_address_` can be directly dereferenced in the current process's virtual address space — i.e., whether `memcpy(dest, buffer_address_, size)` can be executed safely without going through the network. This condition holds **only when the buffer was allocated by the current process itself**. However, the current implementation only checks whether the buffer resides on the same host, not whether it belongs to the same process: ```cpp bool TransferSubmitter::isLocalTransfer( const AllocatedBuffer::Descriptor& handle) const { std::string local_ep = engine_.getLocalIpAndPort(); std::string local_ip = extractIpAddress(local_ep); if (!local_ep.empty()) { std::string handle_ip = extractIpAddress(handle.transport_endpoint_); return !handle.transport_endpoint_.empty() && handle_ip == local_ip; } // Without a local IP we cannot prove locality; disable memcpy. return false; } ``` ### Fix Restore `isLocalTransfer` to compare the full `IP:port` instead of IP only. ### Question Historically the logic did perform a full `IP:port` comparison, but PR #1226 changed it to the current IP-only form. Was there a specific consideration behind that change? Understanding the intent would help confirm that this fix does not introduce any unintended regressions. cc @stmatengss @ykwd

bsakaguchi commented

2026-10-29 21:16:19 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 29, 2026):

Reproduction

Terminal 1

mooncake_master  \
--enable_http_metadata_server=true  \
 --http_metadata_server_host=0.0.0.0  \
 --http_metadata_server_port=8080

Terminal 2

put_worker.py

import numpy as np
import time
from mooncake.store import MooncakeDistributedStore

HOST_IP = "172.17.0.3"          # Your actual IP
METADATA = f"http://{HOST_IP}:8080/metadata"
MASTER   = f"{HOST_IP}:50051"

store = MooncakeDistributedStore()
ret = store.setup(
    f"{HOST_IP}:12345",   
    METADATA,
    4 * 1024**3,          # global_segment_size
    1 * 1024**3,          # local_buffer_size
    "tcp",
    "",      
    MASTER,
    )
assert ret == 0, f"setup failed: {ret}"
print("put_worker ready")

buf = np.random.rand(128 * 1024 * 1024).astype(np.float32)  # 512 MB
i = 0
while True:
    key = f"key_{i % 4}"
    store.put_tensor_from(key, buf.ctypes.data, buf.nbytes)
    print(f"put {key}")
    i += 1

Terminal 3

get_worker.py

import numpy as np
import time
from mooncake.store import MooncakeDistributedStore

HOST_IP = "172.17.0.3"         # Your actual IP
METADATA = f"http://{HOST_IP}:8080/metadata"
MASTER   = f"{HOST_IP}:50051"

store = MooncakeDistributedStore()
ret = store.setup(
    f"{HOST_IP}:12346",   
    METADATA,
    4 * 1024**3,
    1 * 1024**3,
    "tcp",
    "",
    MASTER,
    )
assert ret == 0, f"setup failed: {ret}"
print("get_worker ready, waiting for data...")
time.sleep(3)  

out = np.empty(128 * 1024 * 1024, dtype=np.float32)
i = 0
while True:
    key = f"key_{i % 4}"
    try:
        store.get_tensor_into(key, out.ctypes.data, out.nbytes)
        print(f"got {key}")
    except Exception as e:
        print(f"error: {e}")
    i += 1

Crash Signature

Thread 55 "python3" received signal SIGSEGV, Segmentation fault.

Thread 55 (Thread 0x7fe788ff9640 (LWP 103854) "python3"):
#0 __memmove_evex_unaligned_erms ()
at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
#1 mooncake::MemcpyWorkerPool::workerThread ()
from .../site-packages/mooncake/store.so
#2 std::execute_native_thread_routine
#3 start_thread
#4 clone

@fang-tech commented on GitHub (Apr 29, 2026): ## Reproduction ### Terminal 1 ```bash mooncake_master \ --enable_http_metadata_server=true \ --http_metadata_server_host=0.0.0.0 \ --http_metadata_server_port=8080 ``` ### Terminal 2 put_worker.py ```python import numpy as np import time from mooncake.store import MooncakeDistributedStore HOST_IP = "172.17.0.3" # Your actual IP METADATA = f"http://{HOST_IP}:8080/metadata" MASTER = f"{HOST_IP}:50051" store = MooncakeDistributedStore() ret = store.setup( f"{HOST_IP}:12345", METADATA, 4 * 1024**3, # global_segment_size 1 * 1024**3, # local_buffer_size "tcp", "", MASTER, ) assert ret == 0, f"setup failed: {ret}" print("put_worker ready") buf = np.random.rand(128 * 1024 * 1024).astype(np.float32) # 512 MB i = 0 while True: key = f"key_{i % 4}" store.put_tensor_from(key, buf.ctypes.data, buf.nbytes) print(f"put {key}") i += 1 ``` ### Terminal 3 get_worker.py ```python import numpy as np import time from mooncake.store import MooncakeDistributedStore HOST_IP = "172.17.0.3" # Your actual IP METADATA = f"http://{HOST_IP}:8080/metadata" MASTER = f"{HOST_IP}:50051" store = MooncakeDistributedStore() ret = store.setup( f"{HOST_IP}:12346", METADATA, 4 * 1024**3, 1 * 1024**3, "tcp", "", MASTER, ) assert ret == 0, f"setup failed: {ret}" print("get_worker ready, waiting for data...") time.sleep(3) out = np.empty(128 * 1024 * 1024, dtype=np.float32) i = 0 while True: key = f"key_{i % 4}" try: store.get_tensor_into(key, out.ctypes.data, out.nbytes) print(f"got {key}") except Exception as e: print(f"error: {e}") i += 1 ``` ### Crash Signature Thread 55 "python3" received signal SIGSEGV, Segmentation fault. Thread 55 (Thread 0x7fe788ff9640 (LWP 103854) "python3"): #0 __memmove_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317 #1 mooncake::MemcpyWorkerPool::workerThread () from .../site-packages/mooncake/store.so #2 std::execute_native_thread_routine #3 start_thread #4 clone

bsakaguchi commented

2026-10-29 21:16:20 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 29, 2026):

And if the above fix looks correct, I'd be happy to submit a PR addressing this, along with relevant unit tests.

@fang-tech commented on GitHub (Apr 29, 2026): And if the above fix looks correct, I'd be happy to submit a PR addressing this, along with relevant unit tests.

bsakaguchi commented

2026-10-29 21:16:20 +00:00

Author

Owner

@00fish0 commented on GitHub (Apr 30, 2026):

@fang-tech Thanks a lot for the clean reproduction script and the careful root-cause writeup. I tried it on my side and reproduced the segfault almost immediately with your three-terminal setup — with MC_STORE_MEMCPY=1 the get_worker SIGSEGVs on the first read, and with MC_STORE_MEMCPY=0 it stays alive, exactly as you described.

Your analysis on isLocalTransfer() reads correct to me. The IP-only comparison was introduced in #1226, and combined with the per-process unique ip:port from #852's AutoPortBinder, it can no longer distinguish processes on the same host — so the LOCAL_MEMCPY path ends up dereferencing another process's virtual address. Restoring the full ip:port comparison (as you proposed) is the right fix.

For what it's worth, #1995 was opened independently and seems to land on the same conclusion — it restores the full ip:port comparison in isLocalTransfer(), and additionally makes MemcpyWorkerPool::workerThread() GPU-pointer-aware (a related issue when buffers are cudaMalloc'd). So this issue should be resolved once #1995 merges.

Thanks again for the thorough investigation!

@00fish0 commented on GitHub (Apr 30, 2026): @fang-tech Thanks a lot for the clean reproduction script and the careful root-cause writeup. I tried it on my side and reproduced the segfault almost immediately with your three-terminal setup — with MC_STORE_MEMCPY=1 the get_worker SIGSEGVs on the first read, and with MC_STORE_MEMCPY=0 it stays alive, exactly as you described. Your analysis on isLocalTransfer() reads correct to me. The IP-only comparison was introduced in #1226, and combined with the per-process unique ip:port from #852's AutoPortBinder, it can no longer distinguish processes on the same host — so the LOCAL_MEMCPY path ends up dereferencing another process's virtual address. Restoring the full ip:port comparison (as you proposed) is the right fix. For what it's worth, #1995 was opened independently and seems to land on the same conclusion — it restores the full ip:port comparison in isLocalTransfer(), and additionally makes MemcpyWorkerPool::workerThread() GPU-pointer-aware (a related issue when buffers are cudaMalloc'd). So this issue should be resolved once #1995 merges. Thanks again for the thorough investigation!

bsakaguchi commented

2026-10-29 21:16:20 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 30, 2026):

@00fish0 I’m glad this is helpful.
However I found an additional issue. If we simply revert to a strict IP:port check, for example handle.transport_endpoint_ == engine_.getLocalIpAndPort(), we may unexpectedly fall back from LOCAL_MEMCPY to TRANSFER_ENGINE.

In P2P mode, reverting to IP:port is correct: handle.transport_endpoint_ is derived from engine_.getLocalIpAndPort(), and the endpoint is uniquely bound to the local transfer engine instance.

However, in non-P2P mode, handle.transport_endpoint_ comes from local_hostname_, which is a client initialization parameter and may be a logical name such as localhost. In that case, comparing it against engine_.getLocalIpAndPort() will fail by construction, because the latter is the transfer engine's actual runtime endpoint, for example 192.168.1.23:34127. Even if local_hostname_ is an exact IP address, the comparison can still fail because its port may differ from the port assigned by the transfer engine.

In practice, this means the strict IP:port check almost always fails in non-P2P deployments, causing the path to fall back from LOCAL_MEMCPY to TRANSFER_ENGINE. This may also help explain why #1226 switched back to an IP-only comparison.

@fang-tech commented on GitHub (Apr 30, 2026): @00fish0 I’m glad this is helpful. However I found an additional issue. If we simply revert to a strict `IP:port` check, for example `handle.transport_endpoint_ == engine_.getLocalIpAndPort()`, we may unexpectedly fall back from `LOCAL_MEMCPY` to `TRANSFER_ENGINE`. In P2P mode, reverting to `IP:port` is correct: `handle.transport_endpoint_` is derived from `engine_.getLocalIpAndPort()`, and the endpoint is uniquely bound to the local transfer engine instance. However, in non-P2P mode, `handle.transport_endpoint_` comes from `local_hostname_`, which is a client initialization parameter and may be a logical name such as `localhost`. In that case, comparing it against `engine_.getLocalIpAndPort()` will fail by construction, because the latter is the transfer engine's actual runtime endpoint, for example `192.168.1.23:34127`. Even if `local_hostname_` is an exact IP address, the comparison can still fail because its port may differ from the port assigned by the transfer engine. In practice, this means the strict `IP:port` check almost always fails in non-P2P deployments, causing the path to fall back from `LOCAL_MEMCPY` to `TRANSFER_ENGINE`. This may also help explain why `#1226` switched back to an IP-only comparison.

bsakaguchi commented

2026-10-29 21:16:20 +00:00

Author

Owner

@fang-tech commented on GitHub (Apr 30, 2026):

I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue.

@fang-tech commented on GitHub (Apr 30, 2026): I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue.

bsakaguchi commented

2026-10-29 21:16:20 +00:00

Author

Owner

@00fish0 commented on GitHub (May 6, 2026):

I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue.

Sorry for the late reply! I'll take a look at your PR in the next few days. Thanks for the contribution!

@00fish0 commented on GitHub (May 6, 2026): > I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue. Sorry for the late reply! I'll take a look at your PR in the next few days. Thanks for the contribution!

bsakaguchi referenced this issue

2026-10-29 21:29:08 +00:00

[PR #551] [MERGED] feat(store): add zero copy batch put and get for python binding #1643

Rows
Columns

[GH-ISSUE #1986] [Bug]: SIGSEGV in MemcpyWorkerPool::workerThread under TCP transport with concurrent put/get #551

Bug Report

Summary

Reproduction

Crash signature

Workaround

Environment

Before submitting...

Root Cause Analysis

Fix

Question

Reproduction

Terminal 1

Terminal 2

Terminal 3

Crash Signature

[GH-ISSUE #1986] [Bug]: SIGSEGV in `MemcpyWorkerPool::workerThread` under TCP transport with concurrent put/get #551