kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3329] [Bug]: store_kv_bench --io-api zcopy always fails on non-NOF builds: "direct hugepage alloc failed for zcopy pool" #845

New issue

Closed

opened 2026-10-29 21:20:09 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:20:09 +00:00

Owner

Originally created by @mzygQAQ on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3329

Bug Report

Environment

Mooncake commit: a2966b6a
Build: default CMake (USE_NOF=OFF), store wheel built without SPDK/NoF
Transport: TCP
Store API: --io-api zcopy

Summary
Running mooncake-store/benchmarks/store_kv_bench.py with --io-api zcopy
always fails during session setup on a memory-only (non-NOF) build:

RuntimeError: direct hugepage alloc failed for zcopy pool: size=16384

The benchmark's zcopy path is hard-coupled to the NOF build option, even
though the zero-copy store APIs themselves (put_from / get_into /
batch_put_from / batch_get_into) have nothing to do with NOF and work
fine on memory-only clusters.

Steps to reproduce
Memory-only cluster (1+0), no --nof-replica-num:

python3 mooncake-store/benchmarks/store_kv_bench.py \
  --scenario mixed_rw \
  --prepare-mode auto \
  --io-api zcopy \
  --local-hostname <host>:50071 \
  --metadata-server <metadata-url> \
  --master-server <master:50051> \
  --protocol tcp \
  --nr-objects 64 --write-objects 4096 --batch-size 4 --runtime 10 \
  --rwmixread 70 --key-prefix mixed --key-size 20 --value-size 4096 \
  --memory-replica-num 1 --verify --pattern 0x5a

Note the failure is unrelated to replica count — it triggers before any I/O.

Actual behavior

File "store_kv_bench.py", line 488, in __init__
    self.zcopy_pool = ZcopyBufferPool(...)
File "store_kv_bench.py", line 396, in __init__
    raise RuntimeError(
RuntimeError: direct hugepage alloc failed for zcopy pool: size=16384

Expected behavior
--io-api zcopy should work on non-NOF (memory-only) builds. The zero-copy
APIs are orthogonal to NOF.

Root cause
ZcopyBufferPool in mooncake-store/benchmarks/store_kv_bench.py allocates
its registered buffer pool via get_alloc_func_addr() /
get_free_func_addr() (exported in mooncake-integration/store/store_py.cpp),
which point to hugepage_memory_alloc / hugepage_memory_free in
mooncake-store/src/memory_alloc.cpp:

void *hugepage_memory_alloc(size_t size) {
#ifndef USE_NOF
    return nullptr;   // non-NOF builds always return null
#else
    return mooncake::SpdkWrapper::GetInstance().Alloc(size, 0x1000, -1);
#endif
}

hugepage_memory_alloc only has a real implementation when the store is built
with USE_NOF=ON (default OFF); otherwise it always returns nullptr, so
the zcopy pool construction always fails. This helper was originally added for
the SPDK/NoF worker pool (#2172) and later reused by the benchmark's zcopy
path (#2273), creating an unintended coupling between --io-api zcopy and the
NOF build option.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @mzygQAQ on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3329 ### Bug Report **Environment** - Mooncake commit: `a2966b6a` - Build: default CMake (`USE_NOF=OFF`), store wheel built without SPDK/NoF - Transport: TCP - Store API: `--io-api zcopy` **Summary** Running `mooncake-store/benchmarks/store_kv_bench.py` with `--io-api zcopy` always fails during session setup on a memory-only (non-NOF) build: ``` RuntimeError: direct hugepage alloc failed for zcopy pool: size=16384 ``` The benchmark's zcopy path is hard-coupled to the NOF build option, even though the zero-copy store APIs themselves (`put_from` / `get_into` / `batch_put_from` / `batch_get_into`) have nothing to do with NOF and work fine on memory-only clusters. **Steps to reproduce** Memory-only cluster (1+0), no `--nof-replica-num`: ```bash python3 mooncake-store/benchmarks/store_kv_bench.py \ --scenario mixed_rw \ --prepare-mode auto \ --io-api zcopy \ --local-hostname <host>:50071 \ --metadata-server <metadata-url> \ --master-server <master:50051> \ --protocol tcp \ --nr-objects 64 --write-objects 4096 --batch-size 4 --runtime 10 \ --rwmixread 70 --key-prefix mixed --key-size 20 --value-size 4096 \ --memory-replica-num 1 --verify --pattern 0x5a ``` Note the failure is unrelated to replica count — it triggers before any I/O. **Actual behavior** ``` File "store_kv_bench.py", line 488, in __init__ self.zcopy_pool = ZcopyBufferPool(...) File "store_kv_bench.py", line 396, in __init__ raise RuntimeError( RuntimeError: direct hugepage alloc failed for zcopy pool: size=16384 ``` **Expected behavior** `--io-api zcopy` should work on non-NOF (memory-only) builds. The zero-copy APIs are orthogonal to NOF. **Root cause** `ZcopyBufferPool` in `mooncake-store/benchmarks/store_kv_bench.py` allocates its registered buffer pool via `get_alloc_func_addr()` / `get_free_func_addr()` (exported in `mooncake-integration/store/store_py.cpp`), which point to `hugepage_memory_alloc` / `hugepage_memory_free` in `mooncake-store/src/memory_alloc.cpp`: ```cpp void *hugepage_memory_alloc(size_t size) { #ifndef USE_NOF return nullptr; // non-NOF builds always return null #else return mooncake::SpdkWrapper::GetInstance().Alloc(size, 0x1000, -1); #endif } ``` `hugepage_memory_alloc` only has a real implementation when the store is built with `USE_NOF=ON` (default `OFF`); otherwise it always returns `nullptr`, so the zcopy pool construction always fails. This helper was originally added for the SPDK/NoF worker pool (#2172) and later reused by the benchmark's zcopy path (#2273), creating an unintended coupling between `--io-api zcopy` and the NOF build option. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:20:09 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:20:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 7, 2026):

Thanks for opening this issue, @mzygQAQ!

Field	Value
Issue	#3329
GitHub user ID	`30590325`
Reporter	@mzygQAQ

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 7, 2026): Thanks for opening this issue, @mzygQAQ! | Field | Value | |-------|-------| | **Issue** | #3329 | | **GitHub user ID** | `30590325` | | **Reporter** | @mzygQAQ | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:20:09 +00:00

Author

Owner

@fcczzz commented on GitHub (Aug 7, 2026):

Thanks for the detailed report. I reproduced the allocation failure locally with USE_NOF=OFF.

The root cause is as described: get_alloc_func_addr() is exported in non-NoF builds, but the underlying hugepage_memory_alloc() implementation always returns nullptr. Therefore, ZcopyBufferPool fails before register_buffer() or any Store I/O is reached.

The benchmark should not use this NoF/SPDK-specific allocator for its generic zcopy path. Mooncake already provides the native BufferPool / BufferLease API for zero-copy buffers. It allocates from the Store’s setup-time registered local buffer, which uses normal host memory in non-NoF builds and SPDK DMA memory in NoF builds.

A suitable fix would be to migrate store_kv_bench.py to BufferPool, keep the lease alive for the benchmark lifetime, and require the zcopy pool size to fit within --local-buffer-size unless NoF-safe overflow allocation is explicitly supported.

We should also add a TCP + non-NoF + zcopy regression test.

@fcczzz commented on GitHub (Aug 7, 2026): Thanks for the detailed report. I reproduced the allocation failure locally with USE_NOF=OFF. The root cause is as described: get_alloc_func_addr() is exported in non-NoF builds, but the underlying hugepage_memory_alloc() implementation always returns nullptr. Therefore, ZcopyBufferPool fails before register_buffer() or any Store I/O is reached. The benchmark should not use this NoF/SPDK-specific allocator for its generic zcopy path. Mooncake already provides the native BufferPool / BufferLease API for zero-copy buffers. It allocates from the Store’s setup-time registered local buffer, which uses normal host memory in non-NoF builds and SPDK DMA memory in NoF builds. A suitable fix would be to migrate store_kv_bench.py to BufferPool, keep the lease alive for the benchmark lifetime, and require the zcopy pool size to fit within --local-buffer-size unless NoF-safe overflow allocation is explicitly supported. We should also add a TCP + non-NoF + zcopy regression test.

bsakaguchi commented

2026-10-29 21:20:10 +00:00

Author

Owner

@030611 commented on GitHub (Aug 9, 2026):

I reproduced the code path and prepared a local migration of store_kv_bench.py from the NoF/SPDK hugepage helpers to the native BufferPool / BufferLease API.

The patch keeps one lease alive for the full benchmark lifetime, uses lease.ptr for the existing slot arithmetic, and releases the lease before closing the pool. It also fails early when value_size * slots exceeds --local-buffer-size, so the generic benchmark does not silently depend on overflow registration semantics.

I added a CPU-only regression that imports a store module with BufferPool but no NoF alloc/free helpers, then checks acquisition, slot pointer I/O, cleanup, and the local-buffer capacity guard. The benchmark unit suite passes 7/7, py_compile passes, and git diff --check passes.

Before opening a PR, would you prefer the capacity guard above, or should the benchmark permit BufferPool overflow registration on TCP builds? I have not yet run the full TCP + USE_NOF=OFF integration case on Linux, so I would keep that limitation explicit in the PR.

@030611 commented on GitHub (Aug 9, 2026): I reproduced the code path and prepared a local migration of `store_kv_bench.py` from the NoF/SPDK hugepage helpers to the native `BufferPool` / `BufferLease` API. The patch keeps one lease alive for the full benchmark lifetime, uses `lease.ptr` for the existing slot arithmetic, and releases the lease before closing the pool. It also fails early when `value_size * slots` exceeds `--local-buffer-size`, so the generic benchmark does not silently depend on overflow registration semantics. I added a CPU-only regression that imports a store module with `BufferPool` but no NoF alloc/free helpers, then checks acquisition, slot pointer I/O, cleanup, and the local-buffer capacity guard. The benchmark unit suite passes 7/7, `py_compile` passes, and `git diff --check` passes. Before opening a PR, would you prefer the capacity guard above, or should the benchmark permit `BufferPool` overflow registration on TCP builds? I have not yet run the full TCP + `USE_NOF=OFF` integration case on Linux, so I would keep that limitation explicit in the PR.

bsakaguchi commented

2026-10-29 21:20:10 +00:00

Author

Owner

@mzygQAQ commented on GitHub (Aug 10, 2026):

I reproduced the code path and prepared a local migration of store_kv_bench.py from the NoF/SPDK hugepage helpers to the native BufferPool / BufferLease API.

The patch keeps one lease alive for the full benchmark lifetime, uses lease.ptr for the existing slot arithmetic, and releases the lease before closing the pool. It also fails early when value_size * slots exceeds --local-buffer-size, so the generic benchmark does not silently depend on overflow registration semantics.

I added a CPU-only regression that imports a store module with BufferPool but no NoF alloc/free helpers, then checks acquisition, slot pointer I/O, cleanup, and the local-buffer capacity guard. The benchmark unit suite passes 7/7, py_compile passes, and git diff --check passes.

Before opening a PR, would you prefer the capacity guard above, or should the benchmark permit BufferPool overflow registration on TCP builds? I have not yet run the full TCP + USE_NOF=OFF integration case on Linux, so I would keep that limitation explicit in the PR.

@030611 Sorry, I am just a user of store_kv_bench and haven't read the code carefully, so I can't provide relevant feedback. I suggest you refer to the opinions of other maintainers. Thank you for your PR.

@mzygQAQ commented on GitHub (Aug 10, 2026): > I reproduced the code path and prepared a local migration of `store_kv_bench.py` from the NoF/SPDK hugepage helpers to the native `BufferPool` / `BufferLease` API. > > The patch keeps one lease alive for the full benchmark lifetime, uses `lease.ptr` for the existing slot arithmetic, and releases the lease before closing the pool. It also fails early when `value_size * slots` exceeds `--local-buffer-size`, so the generic benchmark does not silently depend on overflow registration semantics. > > I added a CPU-only regression that imports a store module with `BufferPool` but no NoF alloc/free helpers, then checks acquisition, slot pointer I/O, cleanup, and the local-buffer capacity guard. The benchmark unit suite passes 7/7, `py_compile` passes, and `git diff --check` passes. > > Before opening a PR, would you prefer the capacity guard above, or should the benchmark permit `BufferPool` overflow registration on TCP builds? I have not yet run the full TCP + `USE_NOF=OFF` integration case on Linux, so I would keep that limitation explicit in the PR. @030611 Sorry, I am just a user of store_kv_bench and haven't read the code carefully, so I can't provide relevant feedback. I suggest you refer to the opinions of other maintainers. Thank you for your PR.

bsakaguchi referenced this issue

2026-10-29 21:22:21 +00:00

[GH-ISSUE #4181] [RFC]: [Store] The client heartbeat path must share no lock or thread with data-plane operations #1053

bsakaguchi referenced this issue

2026-10-29 21:30:58 +00:00

[PR #845] [MERGED] feat(store): add client heartbeat support for non ha mode #1865