[PR #1257] [CLOSED] [Feature] Modify nvlink_allocator to enable nvlink intraNode transport in SGlang

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#2112

bsakaguchi commented

2026-10-29 21:32:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1257
Author: @TTThanos
Created: 12/22/2025
Status: ❌ Closed

Base: main ← Head: NewFeature_H20_IntraNode

📝 Commits (9)

3ab0df3 Support Fallback strategy in AllocPinnedMemory in nvlink_transport which supports intraNode nvlink transmission
ddb3a3f Change supportFabricMem logic to support fallback CudaMalloc when CuMemCreate file for Fabric handle
efafeec Refine log print for cudaMalloc
fcafc52 Define Macro to avoid build error in MUSA environment
63e4e38 Add cudaMalloc fallback in nvlink_allocator to support SGlang using intraNode nvlink transfer
04d358b Use precommit to change the code format
bd31338 Merge branch 'kvcache-ai:main' into NewFeature_H20_IntraNode
4a0156b Merge remote-tracking branch 'origin/NewFeature_H20_IntraNode' into NewFeature_H20_IntraNode
75f925b Add log prefix to help identify the error

📊 Changes

2 files changed (+61 additions, -4 deletions)

View changed files

📝 mooncake-transfer-engine/nvlink-allocator/nvlink_allocator.cpp (+56 -0)
📝 mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp (+5 -4)

📄 Description

Description

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

This change is a supplement to the previous PR （https://github.com/kvcache-ai/Mooncake/pull/1234#pullrequestreview-3600667790）, enabling the NVLink allocator to have a fallback mechanism that uses cudaMalloc for GPU memory allocation. This ensures compatibility with SGlang's feature of using NVLink for intra-node communication.

Change has been tested using SGlang

nvlink data volume can be observed when using first 4 H20 cards as Prefill and rest of cards as Decode.

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1257 **Author:** [@TTThanos](https://github.com/TTThanos) **Created:** 12/22/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `NewFeature_H20_IntraNode` --- ### 📝 Commits (9) - [`3ab0df3`](https://github.com/kvcache-ai/Mooncake/commit/3ab0df380b6219d7ada9bb663a812fe8a66e8f78) Support Fallback strategy in AllocPinnedMemory in nvlink_transport which supports intraNode nvlink transmission - [`ddb3a3f`](https://github.com/kvcache-ai/Mooncake/commit/ddb3a3f8d26b655a066783fe6f0bfa96986ab037) Change supportFabricMem logic to support fallback CudaMalloc when CuMemCreate file for Fabric handle - [`efafeec`](https://github.com/kvcache-ai/Mooncake/commit/efafeecf87e7061b1bdcbfb8a73c37ae15ff02a2) Refine log print for cudaMalloc - [`fcafc52`](https://github.com/kvcache-ai/Mooncake/commit/fcafc52442b3ff8da7faae09f0018bbe1798abf2) Define Macro to avoid build error in MUSA environment - [`63e4e38`](https://github.com/kvcache-ai/Mooncake/commit/63e4e38f912fa3da3718460b6793570ca52e7cdb) Add cudaMalloc fallback in nvlink_allocator to support SGlang using intraNode nvlink transfer - [`04d358b`](https://github.com/kvcache-ai/Mooncake/commit/04d358b36374d21c1bee653ec67e30f5118a9449) Use precommit to change the code format - [`bd31338`](https://github.com/kvcache-ai/Mooncake/commit/bd3133898cfffb7724f4548b95dda31caa77f264) Merge branch 'kvcache-ai:main' into NewFeature_H20_IntraNode - [`4a0156b`](https://github.com/kvcache-ai/Mooncake/commit/4a0156be9e2d0b7820b97fd6c97c7cb91f8be849) Merge remote-tracking branch 'origin/NewFeature_H20_IntraNode' into NewFeature_H20_IntraNode - [`75f925b`](https://github.com/kvcache-ai/Mooncake/commit/75f925b54ac679c4f6f4f43dbfb4e6fc67e74491) Add log prefix to help identify the error ### 📊 Changes **2 files changed** (+61 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/nvlink-allocator/nvlink_allocator.cpp` (+56 -0) 📝 `mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp` (+5 -4) </details> ### 📄 Description ## Description  ## Type of Change * Types - [ ] Bug fix - [ ] New feature - [x] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? This change is a supplement to the previous PR （https://github.com/kvcache-ai/Mooncake/pull/1234#pullrequestreview-3600667790）, enabling the NVLink allocator to have a fallback mechanism that uses cudaMalloc for GPU memory allocation. This ensures compatibility with SGlang's feature of using NVLink for intra-node communication. Change has been tested using SGlang <img width="3342" height="618" alt="image" src="https://github.com/user-attachments/assets/fd57c20a-b20b-424f-91e4-f4ea763e4959" /> <img width="2240" height="486" alt="image" src="https://github.com/user-attachments/assets/4ae69897-7902-42c5-a947-b09f837f5b99" /> nvlink data volume can be observed when using first 4 H20 cards as Prefill and rest of cards as Decode. <img width="1560" height="1258" alt="image" src="https://github.com/user-attachments/assets/212b3f08-4201-4a4c-b032-a1126e03008d" /> ## Checklist - [x] I have performed a self-review of my own code. - [x] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

closed this issue
added the
pull-request
label

Rows
Columns

[PR #1257] [CLOSED] [Feature] Modify nvlink_allocator to enable nvlink intraNode transport in SGlang #2112