[GH-ISSUE #4761] [Bug]: [Store] batch_get_into into GPU memory crashes (SIGSEGV) when the local hot cache is enabled #1213

New issue

Closed

opened 2026-10-29 21:24:02 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:24:02 +00:00

Owner

Originally created by @Jackmeson1 on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4761

Bug Report

On main 0e626c5, batch_get_into and batch_get_into_multi_buffers crash the reading process with SIGSEGV when the destination is CUDA memory and the local hot cache is enabled (MC_STORE_LOCAL_HOT_CACHE_SIZE). With the default admission threshold the second read of a key crashes; with MC_STORE_LOCAL_HOT_ADMISSION_THRESHOLD=1 the first one does. The same reads into pinned host memory, get_into into CUDA memory, and the batch read into CUDA memory without the hot cache all succeed.

#0 __memcpy_avx512_unaligned
#1 mooncake::LocalHotCacheHandler::SubmitPutTask(std::string const&, mooncake::Slice const&)
#2 mooncake::Client::ProcessSlicesAsync(...)
#3 mooncake::Client::BatchGet(...)
#4 mooncake::RealClient::batch_get_into_internal(...)
#6 mooncake::RealClient::batch_get_into(...)

The path, at 0e626c5:

batch_get_into hands the caller's buffers, device pointers included, to Client::BatchGet; the multi-buffer path does the same (real_client.cpp:8137).
After the transfer, BatchGet calls ProcessSlicesAsync on those slices (client_service.cpp:1899, and :1646 in BatchGetWhenPreferSameNode), which submits every slice of a remote replica to the hot cache (client_service.cpp:5613-5634).
SubmitPutTask copies the slice with std::memcpy(block->addr, slice.ptr, slice.size), a host copy from a device pointer.
The single-key path avoids this because it stages GPU destinations through host memory whenever IsHotCacheEnabled() (real_client.cpp:4701-4704). The batch paths have no such staging.

Draft #4619 replaces that memcpy with RuntimeAccelerators().CopyToHost (tested with a mocked accelerator), which looks like it would also fix this crash, although its description does not mention it. I'm filing the crash separately so it is tracked with a reproduction on a real GPU, whichever change lands.

Environment

Mooncake main 0e626c5, built with -DUSE_CUDA=ON (HTTP and etcd metadata, unit tests), CUDA 13
NVIDIA RTX PRO 6000 Blackwell (96 GB), Ubuntu, Python 3.13, torch 2.11.0+cu130
One mooncake_master, TCP transport, P2PHANDSHAKE

Steps to reproduce

mooncake_master --default_kv_lease_ttl=500 &
sleep 2                                # let the master start listening on 50051
python repro.py writer > writer.log &  # client without the hot cache puts a 64 KiB key, then waits
until grep -q "put ok" writer.log; do sleep 0.2; done
MC_STORE_LOCAL_HOT_CACHE_SIZE=$((64 << 20)) MC_STORE_LOCAL_HOT_BLOCK_SIZE=$((1 << 20)) python repro.py reader

Condensed from the script behind the results below (same calls and settings):

# repro.py
import os, sys, time
import torch
from mooncake.store import MooncakeDistributedStore

KEY, SIZE, MASTER = "hot_cache_gpu_key", 64 * 1024, "127.0.0.1:50051"
data = bytes((i * 7 + 3) % 251 for i in range(SIZE))
s = MooncakeDistributedStore()

if sys.argv[1] == "writer":
    assert s.setup("localhost:17901", "P2PHANDSHAKE", 256 << 20, 64 << 20, "tcp", "", MASTER) == 0
    assert s.put(KEY, data) == 0
    print("put ok", flush=True)
    time.sleep(600)
else:
    assert s.setup("localhost:17902", "P2PHANDSHAKE", 16 << 20, 64 << 20, "tcp", "", MASTER) == 0
    for attempt in range(3):
        dst = torch.zeros(SIZE, dtype=torch.uint8, device="cuda")
        assert s.register_buffer(dst.data_ptr(), SIZE) == 0
        n = s.batch_get_into([KEY], [dst.data_ptr()], [SIZE])[0]
        torch.cuda.synchronize()
        print(attempt, n, bytes(dst.cpu().numpy()) == data, flush=True)
        time.sleep(0.5)  # as in the recorded runs; the crash does not depend on it
        s.unregister_buffer(dst.data_ptr())

Output: attempt 0 reads the 65536 bytes correctly; the next batch_get_into call then dies with Segmentation fault (exit 139) inside the hot-cache admission copy, before the call returns. The copy runs synchronously in SubmitPutTask, before the fill task is queued.

One recorded run per mode (the first four rows use MC_STORE_LOCAL_HOT_ADMISSION_THRESHOLD=1; the batch CUDA case crashed again under gdb, which gave the backtrace above; with the default threshold, the batch CUDA case crashed on the second read as described):

destination	API	hot cache	result
pinned host	`batch_get_into`	on	3 reads OK
CUDA	`batch_get_into`	off	3 reads OK
CUDA	`get_into`	on	3 reads OK
CUDA	`batch_get_into`	on	SIGSEGV
pinned host	`batch_get_into_multi_buffers`	on	3 reads OK
CUDA	`batch_get_into_multi_buffers`	on	SIGSEGV

Expected behavior

A batch read into GPU memory with the hot cache enabled either fills the cache with a device-aware copy, as get_into effectively does, or skips admission for device slices; it should not crash the process.

This report was prepared with AI assistance (Claude Code); the reproduction was run on a real GPU against 0e626c5.

Originally created by @Jackmeson1 on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4761 ## Bug Report On main [`0e626c5`](https://github.com/kvcache-ai/Mooncake/tree/0e626c572ba3ceee30cef4692bb981487bf65913), `batch_get_into` and `batch_get_into_multi_buffers` crash the reading process with SIGSEGV when the destination is CUDA memory and the local hot cache is enabled (`MC_STORE_LOCAL_HOT_CACHE_SIZE`). With the default admission threshold the second read of a key crashes; with `MC_STORE_LOCAL_HOT_ADMISSION_THRESHOLD=1` the first one does. The same reads into pinned host memory, `get_into` into CUDA memory, and the batch read into CUDA memory without the hot cache all succeed. ``` #0 __memcpy_avx512_unaligned #1 mooncake::LocalHotCacheHandler::SubmitPutTask(std::string const&, mooncake::Slice const&) #2 mooncake::Client::ProcessSlicesAsync(...) #3 mooncake::Client::BatchGet(...) #4 mooncake::RealClient::batch_get_into_internal(...) #6 mooncake::RealClient::batch_get_into(...) ``` The path, at `0e626c5`: - `batch_get_into` hands the caller's buffers, device pointers included, to [`Client::BatchGet`](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/real_client.cpp#L6509); the multi-buffer path does the same ([real_client.cpp:8137](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/real_client.cpp#L8137-L8139)). - After the transfer, `BatchGet` calls `ProcessSlicesAsync` on those slices ([client_service.cpp:1899](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/client_service.cpp#L1899), and [:1646](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/client_service.cpp#L1646) in `BatchGetWhenPreferSameNode`), which submits every slice of a remote replica to the hot cache ([client_service.cpp:5613-5634](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/client_service.cpp#L5613-L5634)). - `SubmitPutTask` copies the slice with [`std::memcpy(block->addr, slice.ptr, slice.size)`](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/local_hot_cache.cpp#L542), a host copy from a device pointer. - The single-key path avoids this because it stages GPU destinations through host memory whenever `IsHotCacheEnabled()` ([real_client.cpp:4701-4704](https://github.com/kvcache-ai/Mooncake/blob/0e626c572ba3ceee30cef4692bb981487bf65913/mooncake-store/src/real_client.cpp#L4701-L4704)). The batch paths have no such staging. Draft #4619 replaces that `memcpy` with `RuntimeAccelerators().CopyToHost` (tested with a mocked accelerator), which looks like it would also fix this crash, although its description does not mention it. I'm filing the crash separately so it is tracked with a reproduction on a real GPU, whichever change lands. ## Environment - Mooncake main `0e626c5`, built with `-DUSE_CUDA=ON` (HTTP and etcd metadata, unit tests), CUDA 13 - NVIDIA RTX PRO 6000 Blackwell (96 GB), Ubuntu, Python 3.13, torch 2.11.0+cu130 - One `mooncake_master`, TCP transport, `P2PHANDSHAKE` ## Steps to reproduce ```bash mooncake_master --default_kv_lease_ttl=500 & sleep 2 # let the master start listening on 50051 python repro.py writer > writer.log & # client without the hot cache puts a 64 KiB key, then waits until grep -q "put ok" writer.log; do sleep 0.2; done MC_STORE_LOCAL_HOT_CACHE_SIZE=$((64 << 20)) MC_STORE_LOCAL_HOT_BLOCK_SIZE=$((1 << 20)) python repro.py reader ``` Condensed from the script behind the results below (same calls and settings): ```python # repro.py import os, sys, time import torch from mooncake.store import MooncakeDistributedStore KEY, SIZE, MASTER = "hot_cache_gpu_key", 64 * 1024, "127.0.0.1:50051" data = bytes((i * 7 + 3) % 251 for i in range(SIZE)) s = MooncakeDistributedStore() if sys.argv[1] == "writer": assert s.setup("localhost:17901", "P2PHANDSHAKE", 256 << 20, 64 << 20, "tcp", "", MASTER) == 0 assert s.put(KEY, data) == 0 print("put ok", flush=True) time.sleep(600) else: assert s.setup("localhost:17902", "P2PHANDSHAKE", 16 << 20, 64 << 20, "tcp", "", MASTER) == 0 for attempt in range(3): dst = torch.zeros(SIZE, dtype=torch.uint8, device="cuda") assert s.register_buffer(dst.data_ptr(), SIZE) == 0 n = s.batch_get_into([KEY], [dst.data_ptr()], [SIZE])[0] torch.cuda.synchronize() print(attempt, n, bytes(dst.cpu().numpy()) == data, flush=True) time.sleep(0.5) # as in the recorded runs; the crash does not depend on it s.unregister_buffer(dst.data_ptr()) ``` Output: attempt 0 reads the 65536 bytes correctly; the next `batch_get_into` call then dies with `Segmentation fault` (exit 139) inside the hot-cache admission copy, before the call returns. The copy runs synchronously in `SubmitPutTask`, before the fill task is queued. One recorded run per mode (the first four rows use `MC_STORE_LOCAL_HOT_ADMISSION_THRESHOLD=1`; the batch CUDA case crashed again under gdb, which gave the backtrace above; with the default threshold, the batch CUDA case crashed on the second read as described): | destination | API | hot cache | result | |---|---|---|---| | pinned host | `batch_get_into` | on | 3 reads OK | | CUDA | `batch_get_into` | off | 3 reads OK | | CUDA | `get_into` | on | 3 reads OK | | CUDA | `batch_get_into` | on | SIGSEGV | | pinned host | `batch_get_into_multi_buffers` | on | 3 reads OK | | CUDA | `batch_get_into_multi_buffers` | on | SIGSEGV | ## Expected behavior A batch read into GPU memory with the hot cache enabled either fills the cache with a device-aware copy, as `get_into` effectively does, or skips admission for device slices; it should not crash the process. This report was prepared with AI assistance (Claude Code); the reproduction was run on a real GPU against `0e626c5`.

bsakaguchi closed this issue

2026-10-29 21:24:02 +00:00

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Thanks for opening this issue, @Jackmeson1!

Field	Value
Issue	#4761
GitHub user ID	`130693642`
Reporter	@Jackmeson1

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Thanks for opening this issue, @Jackmeson1! | Field | Value | |-------|-------| | **Issue** | #4761 | | **GitHub user ID** | `130693642` | | **Reporter** | @Jackmeson1 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 19, 2026):

Thanks for the precise report, @Jackmeson1 — the repro matrix (batch into CUDA crashes, single get_into into CUDA and pinned-host batch reads fine) pointed straight at the right layer.

Root cause: after a successful remote read, Client::ProcessSlicesAsync submits each destination slice to the hot cache via LocalHotCacheHandler::SubmitPutTask, which copies into the cache block with a host std::memcpy(block->addr, slice.ptr, slice.size). For a CUDA destination that memcpy dereferences a device pointer on the host and the process dies in __memcpy_avx512_unaligned. The read side itself is fine — the local-memcpy worker pool is already device-aware, which is why single reads and disabled-cache reads survive.

Fix is up: the writeback now probes each slice with the accelerator registry (the same FindDeviceForPointer discriminator the checksum and DFS staging paths use) and skips device-resident slices instead of caching them. Regression test included: a fake accelerator claims a static buffer as device memory so the guard runs on CPU-only CI, with a host-destined control key in the same batch proving the writeback path itself still works.

→ #4764

@he-yufeng commented on GitHub (Oct 19, 2026): Thanks for the precise report, @Jackmeson1 — the repro matrix (batch into CUDA crashes, single get_into into CUDA and pinned-host batch reads fine) pointed straight at the right layer. Root cause: after a successful remote read, `Client::ProcessSlicesAsync` submits each destination slice to the hot cache via `LocalHotCacheHandler::SubmitPutTask`, which copies into the cache block with a host `std::memcpy(block->addr, slice.ptr, slice.size)`. For a CUDA destination that memcpy dereferences a device pointer on the host and the process dies in `__memcpy_avx512_unaligned`. The read side itself is fine — the local-memcpy worker pool is already device-aware, which is why single reads and disabled-cache reads survive. Fix is up: the writeback now probes each slice with the accelerator registry (the same `FindDeviceForPointer` discriminator the checksum and DFS staging paths use) and skips device-resident slices instead of caching them. Regression test included: a fake accelerator claims a static buffer as device memory so the guard runs on CPU-only CI, with a host-destined control key in the same batch proving the writeback path itself still works. → #4764

bsakaguchi referenced this issue

2026-10-29 21:32:36 +00:00

[PR #1213] [MERGED] [TE] Add TENT codebase to main (Phase 1: structural import) #2077