[PR #1187] [MERGED] [TE] Support rdma traffic class by environmental variable #2059

New issue

Closed

opened 2026-10-29 21:32:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1187
Author: @yafengio
Created: 12/9/2025
Status: ✅ Merged
Merged: 12/10/2025
Merged by: @stmatengss

Base: main ← Head: feat/support-rdma-traffic-class

📝 Commits (4)

dee70ca feat: support RDMA traffic_class by env variable.
04f1757 add some comments and update dump configs.
9ab7e98 add exception details in the log message.
2c27594 code format

📊 Changes

5 files changed (+35 additions, -8 deletions)

View changed files

📝 doc/en/transfer-engine.md (+5 -4)
📝 doc/zh/transfer-engine.md (+5 -4)
📝 mooncake-transfer-engine/include/config.h (+1 -0)
📝 mooncake-transfer-engine/src/config.cpp (+19 -0)
📝 mooncake-transfer-engine/src/transport/rdma_transport/rdma_endpoint.cpp (+5 -0)

📄 Description

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

Description

This PR adds support for configuring RDMA NIC Traffic Class (ToS/DSCP) through the MC_IB_TC environment variable, enabling better network traffic management in production RDMA environments.

Motivation

In production deployments with RDMA networks, different switches and NICs may have inconsistent default Traffic Class configurations. Additionally, network administrators often need to implement traffic planning and QoS policies. Previously, there was no way to adjust the RDMA Traffic Class without modifying code, which limited deployment flexibility.

Changes

1. New Environment Variable

Added MC_IB_TC environment variable support
Valid range: 0-255 (standard IP ToS/DSCP range)
Default value: -1 (use system default, don't set traffic class)

2. Implementation Details

Added ib_traffic_class field to GlobalConfig struct
Parse MC_IB_TC environment variable with proper validation
Range check: 0-255 (uint8_t valid range)
Exception handling for invalid input values
Only set when configured (value >= 0)

4. Documentation Updates

Updated both English and Chinese documentation
- English: [doc/en/transfer-engine.md]
- Chinese: [doc/zh/transfer-engine.md]

Usage Example

# Set Traffic Class to 136
export MC_IB_TC=136

# Run your application
python your_application.py

Compatibility

Backward Compatible: Default value (-1) means no change to existing behavior
No Breaking Changes: Existing deployments continue to work without any modifications

How Has This Been Tested?

This test uses a PD-Disagg deployment (2P4D) and focuses only on the Prefill TTFT value.

Before (Disable MC_IB_TC)

============ Serving Benchmark Result ============
Successful requests:                     5000      
Benchmark duration (s):                  505.93    
Total input tokens:                      20475000  
Total generated tokens:                  5000      
Request throughput (req/s):              9.88      
Output token throughput (tok/s):         9.88      
Total Token throughput (tok/s):          40479.98  
---------------Time to First Token----------------
Mean TTFT (ms):                          1719.96   
Median TTFT (ms):                        1675.05   
P99 TTFT (ms):                           2915.92   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

After(MC_IB_TC=136)

============ Serving Benchmark Result ============
Successful requests:                     5000      
Benchmark duration (s):                  504.56    
Total input tokens:                      20475000  
Total generated tokens:                  5000      
Request throughput (req/s):              9.91      
Output token throughput (tok/s):         9.91      
Total Token throughput (tok/s):          40589.58  
---------------Time to First Token----------------
Mean TTFT (ms):                          1686.23   
Median TTFT (ms):                        1671.34   
P99 TTFT (ms):                           2680.24   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

From the test results above, we can see that the prefill TTFT P99 decreased from 2915.92ms to 2680.24ms, providing a significant improvement (results may vary across different test scenarios).

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1187 **Author:** [@yafengio](https://github.com/yafengio) **Created:** 12/9/2025 **Status:** ✅ Merged **Merged:** 12/10/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `feat/support-rdma-traffic-class` --- ### 📝 Commits (4) - [`dee70ca`](https://github.com/kvcache-ai/Mooncake/commit/dee70ca32032da3449ad79ef7c076631aae8bdd0) feat: support RDMA traffic_class by env variable. - [`04f1757`](https://github.com/kvcache-ai/Mooncake/commit/04f17579d95ef80cf3ce4028b2b7af04aec0e3c4) add some comments and update dump configs. - [`9ab7e98`](https://github.com/kvcache-ai/Mooncake/commit/9ab7e982596ec290c06517a37978d8823ce5dbbd) add exception details in the log message. - [`2c27594`](https://github.com/kvcache-ai/Mooncake/commit/2c27594d8b9cbe102a4ef7fc11946d12d3e27a11) code format ### 📊 Changes **5 files changed** (+35 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/transfer-engine.md` (+5 -4) 📝 `doc/zh/transfer-engine.md` (+5 -4) 📝 `mooncake-transfer-engine/include/config.h` (+1 -0) 📝 `mooncake-transfer-engine/src/config.cpp` (+19 -0) 📝 `mooncake-transfer-engine/src/transport/rdma_transport/rdma_endpoint.cpp` (+5 -0) </details> ### 📄 Description ## Type of Change * Types - [ ] Bug fix - [x] New feature - [x] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## Description This PR adds support for configuring RDMA NIC Traffic Class (ToS/DSCP) through the `MC_IB_TC` environment variable, enabling better network traffic management in production RDMA environments. ## Motivation In production deployments with RDMA networks, different switches and NICs may have inconsistent default Traffic Class configurations. Additionally, network administrators often need to implement traffic planning and QoS policies. Previously, there was no way to adjust the RDMA Traffic Class without modifying code, which limited deployment flexibility. ## Changes #### 1. **New Environment Variable** - Added `MC_IB_TC` environment variable support - **Valid range**: 0-255 (standard IP ToS/DSCP range) - **Default value**: -1 (use system default, don't set traffic class) #### 2. **Implementation Details** - Added `ib_traffic_class` field to `GlobalConfig` struct - Parse `MC_IB_TC` environment variable with proper validation - Range check: 0-255 (uint8_t valid range) - Exception handling for invalid input values - Only set when configured (value >= 0) #### 4. **Documentation Updates** - Updated both English and Chinese documentation - English: [`doc/en/transfer-engine.md`] - Chinese: [`doc/zh/transfer-engine.md`] ### Usage Example ```bash # Set Traffic Class to 136 export MC_IB_TC=136 # Run your application python your_application.py ``` ### Compatibility - **Backward Compatible**: Default value (-1) means no change to existing behavior - **No Breaking Changes**: Existing deployments continue to work without any modifications ## How Has This Been Tested? This test uses a PD-Disagg deployment (2P4D) and focuses only on the Prefill TTFT value. - Before (Disable MC_IB_TC) ``` ============ Serving Benchmark Result ============ Successful requests: 5000 Benchmark duration (s): 505.93 Total input tokens: 20475000 Total generated tokens: 5000 Request throughput (req/s): 9.88 Output token throughput (tok/s): 9.88 Total Token throughput (tok/s): 40479.98 ---------------Time to First Token---------------- Mean TTFT (ms): 1719.96 Median TTFT (ms): 1675.05 P99 TTFT (ms): 2915.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 0.00 Median TPOT (ms): 0.00 P99 TPOT (ms): 0.00 ---------------Inter-token Latency---------------- Mean ITL (ms): 0.00 Median ITL (ms): 0.00 P99 ITL (ms): 0.00 ================================================== ``` - After(MC_IB_TC=136) ``` ============ Serving Benchmark Result ============ Successful requests: 5000 Benchmark duration (s): 504.56 Total input tokens: 20475000 Total generated tokens: 5000 Request throughput (req/s): 9.91 Output token throughput (tok/s): 9.91 Total Token throughput (tok/s): 40589.58 ---------------Time to First Token---------------- Mean TTFT (ms): 1686.23 Median TTFT (ms): 1671.34 P99 TTFT (ms): 2680.24 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 0.00 Median TPOT (ms): 0.00 P99 TPOT (ms): 0.00 ---------------Inter-token Latency---------------- Mean ITL (ms): 0.00 Median ITL (ms): 0.00 P99 ITL (ms): 0.00 ================================================== ``` From the test results above, we can see that the prefill TTFT P99 decreased from 2915.92ms to 2680.24ms, providing a significant improvement (results may vary across different test scenarios). ## Checklist - [x] I have performed a self-review of my own code. - [x] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:27 +00:00