kvcache-ai/Mooncake

Fork

You've already forked Mooncake

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

Code Issues 368 Projects Packages Wiki Activity

[PR #1076] [MERGED] feat: add PCIe Relaxed Ordering (RO) support and RDMA traffic class (… #1995

New issue

Closed

opened 2026-10-29 21:31:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:31:59 +00:00

Owner

Copy link

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1076
Author: @1998zxn
Created: 11/18/2025
Status: ✅ Merged
Merged: 12/15/2025
Merged by: @stmatengss

Base: main ← Head: feature/relaxed-ordering-and-rdma-tc

📝 Commits (7)

eee9562 add PCIe Relaxed Ordering (RO) support.
bf2b5d5 fix: add env variable to control Relaxed Ordering (RO)
81ca67b refactor: simplify logic in mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp
716cbe3 fix(RO): set default value to 0
ab3a83f fix(ci): resolve compilation error in PR build
118e808 Supplement guidance documents
1d7ec64 refactor: remove dlopen, use dlsym alone for symbol checking

📊 Changes

5 files changed (+67 additions, -4 deletions)

View changed files

📝 docs/source/design/transfer-engine/index.md (+1 -0)
📝 docs/source/zh_archive/transfer-engine.md (+1 -0)
📝 mooncake-transfer-engine/include/config.h (+2 -0)
📝 mooncake-transfer-engine/src/config.cpp (+11 -0)
📝 mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp (+52 -4)

📄 Description

…TC) control to improve ordering flexibility and queue-level QoS

Description

This PR optimizes Mooncake’s performance in the 2P1D scenario by introducing two main improvements:

Relaxed Ordering (RO) support to improve PCIe out-of-order handling
RDMA queue selection via environment variable to improve queue-level QoS under burst traffic

These changes effectively reduce KV Cache transfer time, thereby lowering overall TTFT (Time-To-First-Token) latency.

Background

In our deployment scenario using SGLang DeepSeek v3 with 2P1D configuration:

P nodes use tp8, pp2 parallel strategy
D nodes use tp8 parallel strategy

We observed that KV Cache transfers could account for up to 23% of the total TTFT. The reasons are:

Mooncake does not enable Relaxed Ordering by default, reducing PCIe out-of-order packet handling efficiency.
This also addresses the issue discussed in #39.
In 2P1D burst traffic scenarios, RDMA queue scheduling can cause congestion, affecting transfer performance.

By enabling these two features, we reduced TTFT from 650ms to ~585ms, and KV Cache transfer time dropped to 15% of TTFT, showing significant performance improvements.

Implementation Details

Relaxed Ordering (RO) Support

Detects if hardware supports RO and enables it
Added logging to indicate whether RO is enabled during runtime, improving observability

RDMA Queue Selection and QoS

Adds an environment variable to select RDMA queues
Added logging to show the details of traffic classes, helping verify correct functionality

Fully backward compatible; scenarios not using these new features remain unaffected with no regression risk.

Type of Change

Types
- Bug fix
- [*] New feature
  - [*] Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

Checklist

[*] I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1076 **Author:** [@1998zxn](https://github.com/1998zxn) **Created:** 11/18/2025 **Status:** ✅ Merged **Merged:** 12/15/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `feature/relaxed-ordering-and-rdma-tc` --- ### 📝 Commits (7) - [`eee9562`](https://github.com/kvcache-ai/Mooncake/commit/eee9562993a0ad9748fa9613c1f0d63adb86a6cf) add PCIe Relaxed Ordering (RO) support. - [`bf2b5d5`](https://github.com/kvcache-ai/Mooncake/commit/bf2b5d5b12132da384e832032c2c39be7b8de6dc) fix: add env variable to control Relaxed Ordering (RO) - [`81ca67b`](https://github.com/kvcache-ai/Mooncake/commit/81ca67bcf8e613abd8559c8ac1150493385fb393) refactor: simplify logic in mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp - [`716cbe3`](https://github.com/kvcache-ai/Mooncake/commit/716cbe3f2963e69f6ecad78a3841f5925676ea9b) fix(RO): set default value to 0 - [`ab3a83f`](https://github.com/kvcache-ai/Mooncake/commit/ab3a83fc8fcac9afcb6b1f07b00a283a526446c9) fix(ci): resolve compilation error in PR build - [`118e808`](https://github.com/kvcache-ai/Mooncake/commit/118e8086334e7f2ce82bd1bac81cda43a90d22eb) Supplement guidance documents - [`1d7ec64`](https://github.com/kvcache-ai/Mooncake/commit/1d7ec64745c278fd3182bea3be256887760e5442) refactor: remove dlopen, use dlsym alone for symbol checking ### 📊 Changes **5 files changed** (+67 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `docs/source/design/transfer-engine/index.md` (+1 -0) 📝 `docs/source/zh_archive/transfer-engine.md` (+1 -0) 📝 `mooncake-transfer-engine/include/config.h` (+2 -0) 📝 `mooncake-transfer-engine/src/config.cpp` (+11 -0) 📝 `mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp` (+52 -4) </details> ### 📄 Description …TC) control to improve ordering flexibility and queue-level QoS ## Description This PR optimizes Mooncake’s performance in the 2P1D scenario by introducing two main improvements: 1. **Relaxed Ordering (RO) support** to improve PCIe out-of-order handling 2. **RDMA queue selection via environment variable** to improve queue-level QoS under burst traffic These changes effectively reduce KV Cache transfer time, thereby lowering overall TTFT (Time-To-First-Token) latency. ## Background In our deployment scenario using SGLang DeepSeek v3 with 2P1D configuration: - P nodes use tp8, pp2 parallel strategy - D nodes use tp8 parallel strategy We observed that KV Cache transfers could account for up to 23% of the total TTFT. The reasons are: 1. Mooncake does not enable Relaxed Ordering by default, reducing PCIe out-of-order packet handling efficiency. *This also addresses the issue discussed in [#39](https://github.com/kvcache-ai/Mooncake/issues/39).* 2. In 2P1D burst traffic scenarios, RDMA queue scheduling can cause congestion, affecting transfer performance. By enabling these two features, we reduced TTFT from 650ms to ~585ms, and KV Cache transfer time dropped to 15% of TTFT, showing significant performance improvements. ## Implementation Details 1. Relaxed Ordering (RO) Support - Detects if hardware supports RO and enables it - Added logging to indicate whether RO is enabled during runtime, improving observability 2. RDMA Queue Selection and QoS - Adds an environment variable to select RDMA queues - Added logging to show the details of traffic classes, helping verify correct functionality Fully backward compatible; scenarios not using these new features remain unaffected with no regression risk. ## Type of Change * Types - [ ] Bug fix - [*] New feature - [*] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## Checklist - [*] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:31:59 +00:00

closed this issue
added the
pull-request
label

No Branch/Tag specified

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels

Mirrored from GitHub Pull Request

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1995

Reference in a new issue

Repository

kvcache-ai/Mooncake

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns