[PR #393] Add imbalance factor in test_low_latency #547

New issue

Open

opened 2026-10-27 21:03:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/393
Author: @JianboDong
Created: 9/4/2025
Status: 🔄 Open

Base: main ← Head: add_imblance

📝 Commits (5)

bcc90b1 Update test_low_latency.py
212ea44 Update utils.py
1ae1d7f Update test_low_latency.py to solve conflicts
c0c1f76 Update utils.py to solve conflicts
ab2999c Merge branch 'main' into add_imblance

📊 Changes

2 files changed (+433 additions, -37 deletions)

View changed files

📝 tests/test_low_latency.py (+239 -37)
📝 tests/utils.py (+194 -0)

📄 Description

The current test_low_latency script in DeepEP assumes a uniform token distribution across experts (ranks). In real workloads, however, token-to-expert routing is often skewed. To demonstrate this, we recorded token distributions from several layers of the dpsk v3 model taking the MMLU Pro dataset as input. As the figures show, the per rank token counts in individual layers (red line) vary substantially.

While load balancing mechanisms like EPLB can mitigate skew, they cannot perfectly predict future distributions, so meaningful imbalance typically persists. To evaluate DeepEP under these conditions, we propose to extend the test_low_latency script to include imbalanced loads.

Imbalanced Distribution Modeling --distribution
We tried several statistical functions to fit the real distribution. As the following figures demonstrate, the log-normal distribution consistently provides the best approximation of the observed token imbalance(lowest Sum of Squared Errors, SSE). In addition, we also include alternative options, such as gamma and power-law distributions.
Degree of Imbalance: --imbalance-factors
A new command-line argument, --imbalance-factors, is introduced to control the degree of imbalance. This factor is intuitively defined as max_tokens_per_rank / average_tokens_per_rank. This allows users to easily simulate various levels of load skew.
Test Flow:
The benchmark runs a default test (with uniformed distribution) as before. It then proceeds to run several rounds of additional tests with different imbalance factors as specified.
Output Format:
The test results are summarized in tables, with each row shows the results with corresponding imbalance factor (the results with default uniform setting are shown in the first row). Each row includes the target/real imbalance factor, followed by the key metrics such as averaged, max, and min values, across all the ranks. Unlike the per-rank output in original scripts, it provides a more concise, holistic view of the system's overall performance under skewed loads.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/393 **Author:** [@JianboDong](https://github.com/JianboDong) **Created:** 9/4/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `add_imblance` --- ### 📝 Commits (5) - [`bcc90b1`](https://github.com/deepseek-ai/DeepEP/commit/bcc90b1a5c9534e158c82084788f3d3dfa00c75a) Update test_low_latency.py - [`212ea44`](https://github.com/deepseek-ai/DeepEP/commit/212ea443bab6888d94737fa7a0ed364915400149) Update utils.py - [`1ae1d7f`](https://github.com/deepseek-ai/DeepEP/commit/1ae1d7f8304398a8e9af3f1f66a0c9ea80f9138b) Update test_low_latency.py to solve conflicts - [`c0c1f76`](https://github.com/deepseek-ai/DeepEP/commit/c0c1f7626bd65c6ce94fe3ea9b736548dab3cda5) Update utils.py to solve conflicts - [`ab2999c`](https://github.com/deepseek-ai/DeepEP/commit/ab2999c606cb98bcf269c5ed489467101ff51045) Merge branch 'main' into add_imblance ### 📊 Changes **2 files changed** (+433 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `tests/test_low_latency.py` (+239 -37) 📝 `tests/utils.py` (+194 -0) </details> ### 📄 Description The current test_low_latency script in DeepEP assumes a uniform token distribution across experts (ranks). In real workloads, however, token-to-expert routing is often skewed. To demonstrate this, we recorded token distributions from several layers of the dpsk v3 model taking the MMLU Pro dataset as input. As the figures show, the per rank token counts in individual layers (red line) vary substantially. <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/f2ce057c-3462-46f7-aa99-255fb15bcef9" /> <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/e672a53f-fd8f-4791-8e00-e8c24d919c8d" /> <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/58b774ed-90d1-4030-886f-ee83e809122e" /> While load balancing mechanisms like EPLB can mitigate skew, they cannot perfectly predict future distributions, so meaningful imbalance typically persists. To evaluate DeepEP under these conditions, we propose to extend the test_low_latency script to include imbalanced loads. 1. Imbalanced Distribution Modeling --distribution We tried several statistical functions to fit the real distribution. As the following figures demonstrate, the log-normal distribution consistently provides the best approximation of the observed token imbalance(lowest Sum of Squared Errors, SSE). In addition, we also include alternative options, such as gamma and power-law distributions. <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/bffc7dad-48be-444f-a520-c36e3fdec134" /> <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/6443d2df-9138-40f9-a201-d81fb908b6bb" /> <img width="690" height="345" alt="image" src="https://github.com/user-attachments/assets/220c0ea1-426e-40af-94da-b19f001f57fc" /> 2. Degree of Imbalance: --imbalance-factors A new command-line argument, --imbalance-factors, is introduced to control the degree of imbalance. This factor is intuitively defined as max_tokens_per_rank / average_tokens_per_rank. This allows users to easily simulate various levels of load skew. 3. Test Flow: The benchmark runs a default test (with uniformed distribution) as before. It then proceeds to run several rounds of additional tests with different imbalance factors as specified. 4. Output Format: The test results are summarized in tables, with each row shows the results with corresponding imbalance factor (the results with default uniform setting are shown in the first row). Each row includes the target/real imbalance factor, followed by the key metrics such as averaged, max, and min values, across all the ranks. Unlike the per-rank output in original scripts, it provides a more concise, holistic view of the system's overall performance under skewed loads. <img width="690" height="605" alt="image" src="https://github.com/user-attachments/assets/eb2cb7bd-cf2d-4174-8b30-bc3cd98c285b" /> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:03:53 +00:00

bsakaguchi referenced this issue

2026-10-27 21:04:41 +00:00

[PR #547] Improve .gitignore with common Python entries #630