[GH-ISSUE #3503] [TE][UB]: Auto-chunk registrations larger than max_seg_size and split cross-segment transfers #896

New issue

Open

opened 2026-10-29 21:20:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:39 +00:00

Owner

Originally created by @staryxchen on GitHub (Aug 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3503

Parent: #3500

Problem

UB is behind both EFA/CXI and RDMA.

Registration still shrinks. UrmaContext::registerMemoryRegion clamps length to max_seg_size instead of splitting into multiple segments. Metadata can advertise more bytes than were actually registered — the same class of bug #2644 fixed for RDMA.
Submit path does not cap at BufferDesc seams. UbTransport::submitTransferTask slices by slice_size only. The initial selectDevice uses the full request.length, so a request that spans two registered segments fails even if individual slices would fit.

Until both are fixed, Store must keep splitting UB segments at max_seg_size.

Proposed work

Auto-chunk registerLocalMemory into <= max_seg_size segments and publish one BufferDesc per chunk (do not shrink).
Cap submit-time slices at source and target BufferDesc remainder, similar to RDMA SliceLengthCalculator.

Test plan

Needs Kunpeng / UB hardware. Suggested coverage:

Register a buffer larger than max_seg_size and confirm it is not truncated
WRITE whose source or destination straddles a chunk seam

Originally created by @staryxchen on GitHub (Aug 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3503 Parent: #3500 ## Problem UB is behind both EFA/CXI and RDMA. 1. **Registration still shrinks.** `UrmaContext::registerMemoryRegion` clamps `length` to `max_seg_size` instead of splitting into multiple segments. Metadata can advertise more bytes than were actually registered — the same class of bug #2644 fixed for RDMA. 2. **Submit path does not cap at `BufferDesc` seams.** `UbTransport::submitTransferTask` slices by `slice_size` only. The initial `selectDevice` uses the full `request.length`, so a request that spans two registered segments fails even if individual slices would fit. Until both are fixed, Store must keep splitting UB segments at `max_seg_size`. ## Proposed work 1. Auto-chunk `registerLocalMemory` into `<= max_seg_size` segments and publish one `BufferDesc` per chunk (do not shrink). 2. Cap submit-time slices at source and target `BufferDesc` remainder, similar to RDMA `SliceLengthCalculator`. ## Test plan Needs Kunpeng / UB hardware. Suggested coverage: - Register a buffer larger than `max_seg_size` and confirm it is not truncated - WRITE whose source or destination straddles a chunk seam

bsakaguchi added the

Transfer Engine

feature request

labels

2026-10-29 21:20:39 +00:00

bsakaguchi commented

2026-10-29 21:20:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 18, 2026):

Thanks for opening this issue, @staryxchen!

Field	Value
Issue	#3503
GitHub user ID	`151037142`
Reporter	@staryxchen

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 18, 2026): Thanks for opening this issue, @staryxchen! | Field | Value | |-------|-------| | **Issue** | #3503 | | **GitHub user ID** | `151037142` | | **Reporter** | @staryxchen | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:08 +00:00

[PR #896] [MERGED] [Store]: Get start_time before calling RPC in BatchQuery #1889