UB #897

New issue

Open

opened 2026-10-29 21:20:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:39 +00:00

Owner

Originally created by @staryxchen on GitHub (Aug 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3500

Background

RDMA Transfer Engine already handles buffers larger than the device MR limit end-to-end:

Registration auto-chunks into multiple MRs / BufferDescs (#2644).
Submit-time slicing caps each WR at the source and target BufferDesc remainder (SliceLengthCalculator in RdmaTransport::submitTransferTask), so a single TE read/write that straddles a chunk seam still works.

Because of that, Mooncake Store no longer needs to split RDMA segments at max_mr_size. #3486 is dropping rdma from GetTransportRegistrationLimit() and leaving the Store-side split in place for EFA, CXI, and UB.

Goal

Bring EFA, CXI, and UB to the same TE behavior as RDMA, so Store only needs a registration-size split for transports that still cannot register or transfer across chunk seams.

After all three sub-issues land, GetTransportRegistrationLimit() should only apply to transports that still lack this support (today that is expected to be none, or UB until its registration shrink is fixed).

Why this is split into sub-issues

I do not have EFA or CXI hardware to implement and verify the submit-time path, and UB is a different gap (it still shrinks oversized registrations instead of chunking). Tracking them separately keeps owners and test requirements clear.

Sub-issues

#3501 EFA: submit-time cross-BufferDesc slicing (registration chunking already exists)
#3502 CXI: submit-time cross-BufferDesc slicing (registration chunking already exists)
#3503 UB: registration auto-chunking and submit-time cross-BufferDesc slicing

References

#2644 RDMA registration auto-chunk
#3486 Store protocol-aware segment split
RDMALargeMrTest.WriteStraddlesChunkBoundary / WriteWithSourceStraddlingChunkBoundary

Originally created by @staryxchen on GitHub (Aug 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3500 ## Background RDMA Transfer Engine already handles buffers larger than the device MR limit end-to-end: 1. Registration auto-chunks into multiple MRs / `BufferDesc`s (#2644). 2. Submit-time slicing caps each WR at the source and target `BufferDesc` remainder (`SliceLengthCalculator` in `RdmaTransport::submitTransferTask`), so a single TE read/write that straddles a chunk seam still works. Because of that, Mooncake Store no longer needs to split RDMA segments at `max_mr_size`. #3486 is dropping `rdma` from `GetTransportRegistrationLimit()` and leaving the Store-side split in place for EFA, CXI, and UB. ## Goal Bring EFA, CXI, and UB to the same TE behavior as RDMA, so Store only needs a registration-size split for transports that still cannot register or transfer across chunk seams. After all three sub-issues land, `GetTransportRegistrationLimit()` should only apply to transports that still lack this support (today that is expected to be none, or UB until its registration shrink is fixed). ## Why this is split into sub-issues I do not have EFA or CXI hardware to implement and verify the submit-time path, and UB is a different gap (it still shrinks oversized registrations instead of chunking). Tracking them separately keeps owners and test requirements clear. ## Sub-issues - [x] #3501 EFA: submit-time cross-`BufferDesc` slicing (registration chunking already exists) - [ ] #3502 CXI: submit-time cross-`BufferDesc` slicing (registration chunking already exists) - [ ] #3503 UB: registration auto-chunking **and** submit-time cross-`BufferDesc` slicing ## References - #2644 RDMA registration auto-chunk - #3486 Store protocol-aware segment split - `RDMALargeMrTest.WriteStraddlesChunkBoundary` / `WriteWithSourceStraddlingChunkBoundary`

bsakaguchi added the

Transfer Engine

feature request

labels

2026-10-29 21:20:39 +00:00

bsakaguchi commented

2026-10-29 21:20:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 18, 2026):

Thanks for opening this issue, @staryxchen!

Field	Value
Issue	#3500
GitHub user ID	`151037142`
Reporter	@staryxchen

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 18, 2026): Thanks for opening this issue, @staryxchen! | Field | Value | |-------|-------| | **Issue** | #3500 | | **GitHub user ID** | `151037142` | | **Reporter** | @staryxchen | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:09 +00:00

[PR #897] [MERGED] docs(store): restructure and simplify SGLang HiCache integration guide #1890