[GH-ISSUE #4883] [Bug]: Direct NVMe-oF requests past segment end can report partial I/O as COMPLETED #1242

New issue

Open

opened 2026-10-29 21:24:19 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:19 +00:00

Owner

Originally created by @RuiqingFeng on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4883

Environment and affected path

Mooncake upstream main c67a88f381
(2026-10-23). This concerns the classic Transfer Engine's direct
NVMeoFTransport::submitTransfer(batch_id, entries) path, not Store's SPDK
NoF path or NVMe KV backend. The MultiTransport task-batch path is a
separate, explicitly unsupported interface for this transport.

Reproduction

Create an NVMe-oF segment with two 4096-byte buffers. Allocate a direct
transport batch and submit a WRITE with target_offset=4096 and length=8192.
The segment is only 8192 bytes long, so half the request lies outside it.

Observed with the real transport and descriptor-pool sources linked to fake
cuFile entry points on Linux (GCC 13.3, C++20):

submitTransfer returns OK and submits one 4096-byte cuFile operation.
Once that submitted slice completes, getTransferStatus reports COMPLETED
with transferred_bytes=4096, although requested_bytes=8192.
A submission containing a valid request followed by an invalid request
also mutates the batch and submits I/O instead of rejecting the call.

The fake backend records submitted cuFile parameters and completes exactly
those operations; no real GPU, NVMe device, or storage DMA was exercised.
Two legal single-/cross-buffer control cases pass on the same baseline.

Expected behavior

Reject a request that is not fully covered by the segment with
InvalidArgument before creating any tasks, slices, file handles, or I/O
parameters for that submission. Check every request before mutating the
batch, retain any existing tasks, and permit a subsequent valid submission.
Use overflow-safe arithmetic for the segment length and request range.
This is validation atomicity, not transactional I/O after successful submit.

Cause

submitTransfer intersects each request with nvmeof_buffers and only submits
those intersections. It never verifies that the requested interval is fully
covered. aggregateTransferStatus only aggregates submitted slices, so it
cannot detect bytes that were omitted before submission. The request-end
addition and cumulative buffer lengths are also unchecked.

I searched upstream issues and PRs using the authenticated GitHub API.
#2893 and #2939 address task submission/completion correlation and terminal
failure propagation. Open PR #2917 changes cancellation/draining and
submission-error rollback; its submitTransfer diff still uses the existing
intersection-only slicing without full request-range validation. This report
covers pre-submission range validation, not that lifecycle work.

I have prepared a focused fix and regression tests for full-range
prevalidation, integer overflow, mixed-validity submissions, retry and
existing-task preservation. I would like to contribute the fix.

AI assistance

Codex assisted with investigation, implementation and test preparation.

Originally created by @RuiqingFeng on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4883 ## Environment and affected path Mooncake upstream main c67a88f381f69e869045f353539e90c76e738830 (2026-10-23). This concerns the classic Transfer Engine's direct NVMeoFTransport::submitTransfer(batch_id, entries) path, not Store's SPDK NoF path or NVMe KV backend. The MultiTransport task-batch path is a separate, explicitly unsupported interface for this transport. ## Reproduction Create an NVMe-oF segment with two 4096-byte buffers. Allocate a direct transport batch and submit a WRITE with target_offset=4096 and length=8192. The segment is only 8192 bytes long, so half the request lies outside it. Observed with the real transport and descriptor-pool sources linked to fake cuFile entry points on Linux (GCC 13.3, C++20): - submitTransfer returns OK and submits one 4096-byte cuFile operation. - Once that submitted slice completes, getTransferStatus reports COMPLETED with transferred_bytes=4096, although requested_bytes=8192. - A submission containing a valid request followed by an invalid request also mutates the batch and submits I/O instead of rejecting the call. The fake backend records submitted cuFile parameters and completes exactly those operations; no real GPU, NVMe device, or storage DMA was exercised. Two legal single-/cross-buffer control cases pass on the same baseline. ## Expected behavior Reject a request that is not fully covered by the segment with InvalidArgument before creating any tasks, slices, file handles, or I/O parameters for that submission. Check every request before mutating the batch, retain any existing tasks, and permit a subsequent valid submission. Use overflow-safe arithmetic for the segment length and request range. This is validation atomicity, not transactional I/O after successful submit. ## Cause submitTransfer intersects each request with nvmeof_buffers and only submits those intersections. It never verifies that the requested interval is fully covered. aggregateTransferStatus only aggregates submitted slices, so it cannot detect bytes that were omitted before submission. The request-end addition and cumulative buffer lengths are also unchecked. ## Related work and proposed fix I searched upstream issues and PRs using the authenticated GitHub API. #2893 and #2939 address task submission/completion correlation and terminal failure propagation. Open PR #2917 changes cancellation/draining and submission-error rollback; its submitTransfer diff still uses the existing intersection-only slicing without full request-range validation. This report covers pre-submission range validation, not that lifecycle work. I have prepared a focused fix and regression tests for full-range prevalidation, integer overflow, mixed-validity submissions, retry and existing-task preservation. I would like to contribute the fix. ## AI assistance Codex assisted with investigation, implementation and test preparation.

bsakaguchi commented

2026-10-29 21:24:20 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 23, 2026):

Thanks for opening this issue, @RuiqingFeng!

Field	Value
Issue	#4883
GitHub user ID	`293799895`
Reporter	@RuiqingFeng

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 23, 2026): Thanks for opening this issue, @RuiqingFeng! | Field | Value | |-------|-------| | **Issue** | #4883 | | **GitHub user ID** | `293799895` | | **Reporter** | @RuiqingFeng | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:33:10 +00:00

[PR #1313] [MERGED] [EP] Replace _mm_pause() with PAUSE() macro #2150