[PR #581] [MERGED] feat(topology): improve HCA selection by considering PCIe distance #1667

New issue

Closed

opened 2026-10-29 21:29:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:20 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/581
Author: @staryxchen
Created: 7/2/2025
Status: ✅ Merged
Merged: 7/8/2025
Merged by: @wx-csy

Base: main ← Head: bugfix/distancecalc

📝 Commits (1)

b4ca77d feat(topology): improve HCA selection by considering PCIe distance

📊 Changes

1 file changed (+20 additions, -3 deletions)

View changed files

📝 mooncake-transfer-engine/src/topology.cpp (+20 -3)

📄 Description

Background

The function getPciDistance is used to determine the PCIe topological distance between two devices (e.g., GPU and NIC) by comparing their sysfs paths. Previously, the function only returned 0 if the two devices had exactly the same sysfs path prefix, i.e., their paths were identical except for the device itself. This approach failed to recognize that two devices under the same PCIe switch/root port, but with different device IDs, should also be considered as having a distance of 0.

Problem Example

Suppose we have two PCIe devices with the following bus IDs and sysfs paths:

bus1: 0000:bc:00.0, path1: /sys/devices/pci0000:b2/0000:b2:03.1/0000:b9:00.0/0000:ba:01.0/0000:bc:00.0
bus2: 0000:bb:00.0, path2: /sys/devices/pci0000:b2/0000:b2:03.1/0000:b9:00.0/0000:ba:00.0/0000:bb:00.0

In this case, both devices are under the same PCIe switch, but their sysfs paths differ only in the last component (the device itself). The previous implementation would not recognize them as being under the same switch and would return a non-zero distance.

Fix

This patch improves getPciDistance by first removing the last component (the device itself) from both sysfs paths, and then applying the original distance calculation algorithm. If the parent paths are identical, the function now correctly returns 0, indicating that the two devices are under the same PCIe switch/root port.

Impact

More accurate PCIe topology awareness: Devices under the same switch are now correctly treated as having distance 0.
Better NIC/GPU affinity: This improves the logic for selecting preferred NICs for GPUs, especially in multi-GPU/multi-NIC systems.
Backward compatibility: The fallback logic for non-matching parent paths remains unchanged.

If you have any questions about the patch or need more real-world examples, feel free to ask!

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/581 **Author:** [@staryxchen](https://github.com/staryxchen) **Created:** 7/2/2025 **Status:** ✅ Merged **Merged:** 7/8/2025 **Merged by:** [@wx-csy](https://github.com/wx-csy) **Base:** `main` ← **Head:** `bugfix/distancecalc` --- ### 📝 Commits (1) - [`b4ca77d`](https://github.com/kvcache-ai/Mooncake/commit/b4ca77d54e39c3aab27363dfa9ab0a37d48f7f10) feat(topology): improve HCA selection by considering PCIe distance ### 📊 Changes **1 file changed** (+20 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/src/topology.cpp` (+20 -3) </details> ### 📄 Description # Background The function `getPciDistance` is used to determine the PCIe topological distance between two devices (e.g., GPU and NIC) by comparing their sysfs paths. Previously, the function only returned 0 if the two devices had exactly the same sysfs path prefix, i.e., their paths were identical except for the device itself. This approach failed to recognize that two devices under the same PCIe switch/root port, but with different device IDs, should also be considered as having a distance of 0. # Problem Example Suppose we have two PCIe devices with the following bus IDs and sysfs paths: - bus1: `0000:bc:00.0`, path1: `/sys/devices/pci0000:b2/0000:b2:03.1/0000:b9:00.0/0000:ba:01.0/0000:bc:00.0` - bus2: `0000:bb:00.0`, path2: `/sys/devices/pci0000:b2/0000:b2:03.1/0000:b9:00.0/0000:ba:00.0/0000:bb:00.0` In this case, both devices are under the same PCIe switch, but their sysfs paths differ only in the last component (the device itself). The previous implementation would not recognize them as being under the same switch and would return a non-zero distance. # Fix This patch improves `getPciDistance` by first removing the last component (the device itself) from both sysfs paths, and then applying the original distance calculation algorithm. If the parent paths are identical, the function now correctly returns 0, indicating that the two devices are under the same PCIe switch/root port. # Impact - More accurate PCIe topology awareness: Devices under the same switch are now correctly treated as having distance 0. - Better NIC/GPU affinity: This improves the logic for selecting preferred NICs for GPUs, especially in multi-GPU/multi-NIC systems. - Backward compatibility: The fallback logic for non-matching parent paths remains unchanged. If you have any questions about the patch or need more real-world examples, feel free to ask! --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:20 +00:00