[GH-ISSUE #4029] [Bug]: NoF client cannot handle unaligned buffer lengths and addresses #1018

New issue

Open

opened 2026-10-29 21:22:03 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:22:03 +00:00

Owner

Originally created by @cage-goat on GitHub (Sep 11, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4029

Bug Report

Related to #4021 and PR #4024.

#4024 addresses master-side disk allocation alignment and reserves padding space for objects whose sizes are not multiples of the namespace block size. For example, with block_size=512 B, a 513 B object receives at least 1024 B of disk space, including the 511 B needed to complete the last LBA. Its logical size remains 513 B.

This additional space is reserved on disk; it does not extend the caller's memory buffer. The client therefore needs to construct a 1024 B I/O without reading or writing beyond the caller's 513 B buffer.

Upstream applications such as LMCache and SGLang pass logical object sizes, which are not necessarily multiples of the namespace block_size. The actual buffer address may also be unaligned. Currently, the client rejects these requests when its alignment checks fail.

The client should adapt I/O lengths and buffer layouts while preserving logical object sizes, minimizing payload copies, and allowing memcpy when necessary.

The RDMA path should use SGLs to describe the data. The NVMe base specification distinguishes the following SGL capabilities:

Support level	Meaning
SGL unsupported	The corresponding SGL data-description path is unavailable
SGL supported with DWORD alignment	Data-segment addresses and lengths must be 4-byte aligned
Byte-granular SGL supported	No additional alignment requirements for data-segment addresses or lengths

The client should select a strategy based on the usable transport capabilities, buffer layout, and memory registration, rather than SGL support alone:

Direct SGL transfer: Describe the original buffer and separate padding. Use zero padding for writes and a separate writable padding buffer for reads, without copying valid payload data.
Tail-only copy: When the complete-LBA prefix can be transferred directly, copy only the remaining valid bytes into a temporary tail block. For reads, copy only the valid tail bytes back after completion.
Full payload copy: When the prefix cannot be transferred directly because of its address, layout, or registration, use a suitable staging buffer. Process the object in chunks if needed to bound temporary memory usage.

For example, if the first 512 B of a 513 B object can be transferred directly, only the final byte needs to be copied into a zero-padded 512 B tail block. The prefix and tail block can form a valid SGL or be submitted separately.

A tail-only copy cannot fix an incompatible starting address. If the selected path requires 4-byte alignment but the buffer starts at base + 1, full staging is needed. A byte-granular path should not introduce copies solely because the address is not 4-byte aligned.

The TCP path should use local iovecs to pad the I/O length. SPDK readv/writev can describe:

Writes: original data plus separate zero padding.
Reads: the original destination plus a separate padding buffer.

The total I/O length should satisfy LBA alignment while the logical object size remains unchanged. Callers should not need to extend their buffers. Prefer direct use of the original buffer, with temporary buffers and necessary copies when direct use is unavailable.

Expected results:

Support valid inputs with non-block-multiple lengths and unaligned memory addresses on namespaces with 512 B and 4096 B block sizes, without accessing memory beyond the caller's buffer.
Preserve correct readback data, logical object sizes, returned lengths, and checksums.
Cover single, batch, and multi-buffer operations without adding unnecessary payload copies to requests that can already be transferred directly.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @cage-goat on GitHub (Sep 11, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4029 ### Bug Report Related to [#4021](https://github.com/kvcache-ai/Mooncake/issues/4021) and [PR #4024](https://github.com/kvcache-ai/Mooncake/pull/4024). #4024 addresses master-side disk allocation alignment and reserves padding space for objects whose sizes are not multiples of the namespace block size. For example, with `block_size=512 B`, a 513 B object receives at least 1024 B of disk space, including the 511 B needed to complete the last LBA. Its logical size remains 513 B. This additional space is reserved on disk; it does not extend the caller's memory buffer. The client therefore needs to construct a 1024 B I/O without reading or writing beyond the caller's 513 B buffer. Upstream applications such as LMCache and SGLang pass logical object sizes, which are not necessarily multiples of the namespace `block_size`. The actual buffer address may also be unaligned. Currently, the client rejects these requests when its alignment checks fail. The client should adapt I/O lengths and buffer layouts while preserving logical object sizes, minimizing payload copies, and allowing `memcpy` when necessary. **The RDMA path should use SGLs to describe the data.** The NVMe base specification distinguishes the following SGL capabilities: | Support level | Meaning | |---|---| | SGL unsupported | The corresponding SGL data-description path is unavailable | | SGL supported with DWORD alignment | Data-segment addresses and lengths must be 4-byte aligned | | Byte-granular SGL supported | No additional alignment requirements for data-segment addresses or lengths | The client should select a strategy based on the usable transport capabilities, buffer layout, and memory registration, rather than SGL support alone: - **Direct SGL transfer:** Describe the original buffer and separate padding. Use zero padding for writes and a separate writable padding buffer for reads, without copying valid payload data. - **Tail-only copy:** When the complete-LBA prefix can be transferred directly, copy only the remaining valid bytes into a temporary tail block. For reads, copy only the valid tail bytes back after completion. - **Full payload copy:** When the prefix cannot be transferred directly because of its address, layout, or registration, use a suitable staging buffer. Process the object in chunks if needed to bound temporary memory usage. For example, if the first 512 B of a 513 B object can be transferred directly, only the final byte needs to be copied into a zero-padded 512 B tail block. The prefix and tail block can form a valid SGL or be submitted separately. A tail-only copy cannot fix an incompatible starting address. If the selected path requires 4-byte alignment but the buffer starts at `base + 1`, full staging is needed. A byte-granular path should not introduce copies solely because the address is not 4-byte aligned. **The TCP path should use local iovecs to pad the I/O length.** SPDK `readv/writev` can describe: - Writes: original data plus separate zero padding. - Reads: the original destination plus a separate padding buffer. The total I/O length should satisfy LBA alignment while the logical object size remains unchanged. Callers should not need to extend their buffers. Prefer direct use of the original buffer, with temporary buffers and necessary copies when direct use is unavailable. Expected results: - Support valid inputs with non-block-multiple lengths and unaligned memory addresses on namespaces with 512 B and 4096 B block sizes, without accessing memory beyond the caller's buffer. - Preserve correct readback data, logical object sizes, returned lengths, and checksums. - Cover single, batch, and multi-buffer operations without adding unnecessary payload copies to requests that can already be transferred directly. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:22:03 +00:00

bsakaguchi commented

2026-10-29 21:22:04 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 11, 2026):

Thanks for opening this issue, @cage-goat!

Field	Value
Issue	#4029
GitHub user ID	`318081269`
Reporter	@cage-goat

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 11, 2026): Thanks for opening this issue, @cage-goat! | Field | Value | |-------|-------| | **Issue** | #4029 | | **GitHub user ID** | `318081269` | | **Reporter** | @cage-goat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:41 +00:00

[PR #1018] [CLOSED] Add C++ builder pattern #1957