[GH-ISSUE #4681] [RFC]: [TENT] Runtime-loadable transport plugins #1193

New issue

Open

opened 2026-10-29 21:23:48 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:48 +00:00

Owner

Originally created by @xiaodouzi666 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4681

Changes proposed

Revised after maintainer review: the design below is the reduced one now in #4776 and #4777. The first version of this RFC (ABI fingerprint, reserved enum slots, a name table and wire-format changes, a registry with outcome codes, an SDK, a conformance suite and an inspect tool) is withdrawn; #4778 is closed.

Motivation

Every TENT transport is compiled into the core. Roadmap #1058 ("Packaging & Usability: enable runtime dynamic loading") asks for a transport to be buildable and shippable apart from the TENT that uses it, so that a device or vendor backend can be packaged on its own, and for a missing or unusable plugin to be diagnosed clearly.

Design

Interface (include/tent/transport_plugin.h, one header). A plugin is a shared library libtent_xport_<name>.so that exports two functions. tent_transport_plugin_build_id() is plain C and returns the build ID the plugin was compiled with: the source revision and the compiler, which CMake records in a generated tent/build_id.h when the build is configured, plus the libstdc++ string ABI (_GLIBCXX_USE_CXX11_ABI). tent_register_transport_plugin() (the same word order as tent_register_device_plugin) returns a descriptor with the TransportType it provides and create(context), which receives the engine's config and topology and returns a Transport (or nullptr when the plugin does not apply to that engine). A plugin uses TENT's C++ classes directly, so it only runs in the build it was compiled in; the build ID is what TENT checks, through the C function, before it calls any of the plugin's C++ code (as asked in the review of #4776).

Identity. A plugin provides one of the existing TransportType values. The enum, the metadata format, the selector and every transport are unchanged. A new kind of transport still adds its enumerator to TENT, as today.

Loading (src/runtime/transport_plugin_loader.cpp, one call at the end of loadTransports()). Nothing is loaded unless MC_TENT_PLUGIN_DIR or the plugins/dir configuration key names a directory. Files named libtent_xport_*.so are opened in name order. A plugin from another build, or one without a build ID, stops the program with a fatal error (LOG(FATAL), like TENT's other configuration errors) that names the plugin and, for a mismatch, both build IDs. Otherwise a plugin only fills a slot the core left empty; a file that cannot be used (it cannot be opened, has no entry point, names an unknown type or a slot already provided, or its create() throws or declines) is skipped with one log line; plugins are never unloaded. A plugin links no TENT library and resolves TENT's symbols in the process: an executable that links TENT statically must export them, and a TENT inside an object loaded RTLD_LOCAL (a Python extension) is promoted to the global scope before the first plugin is opened.

Built-in transports as plugins (#4777). TENT_XPORT_TCP and TENT_XPORT_RDMA choose static (the default, unchanged) or plugin, in which case TENT does not create the transport and libtent_xport_tcp.so / libtent_xport_rdma.so are built from the sources of the transport's static library plus a short entry file; they create their transport under the same conditions as the built-in one (transports/<name>/enable, and an RDMA NIC in the topology). In plugin mode TENT's tests run against the plugins, including the Python guard test, and a tent-ci leg (xport-plugin) builds and tests this mode.

Size

#4776: +918 lines over 12 files (product 309, tests 504, docs 105). #4777: +215/−1 over 9 files, including 6 lines of CI. No existing transport, enum value, metadata field or public API changes.

Measurements

On one 8×H20 host, RDMA mlx5_4 → mlx5_2, 1–8 threads × 4 KiB–64 MiB, read throughput as the median of 3 runs: the plugin build is within 3% of the static build at all 60 points (−2.52% to +0.31%). Below 4 MiB it is up to 2.5% slower, mostly with 1–4 threads; from 4 MiB up the two builds are within 1%, and with two or more threads both reach about 49 GB/s at 8 MiB and above. The read/write mix is consistent at 60/60 points in both builds, and the same plugin-mode binary loads RDMA only when the plugin directory is set. (Measured on an earlier revision of the branches. Since then the loader's checks were reordered and the build ID check was added, and the RDMA plugin changed only in the name of its entry symbol, in how its sources are listed and by the function that returns its build ID.)

Rollout

PR 1 (#4776): interface, loader, tests, docs.
PR 2 (#4777): TCP and RDMA as plugins, stacked on PR 1.
#4778 (conformance suite, tent_plugin_inspect): closed.

Limitations and possible follow-ups

Same build required: the build ID records the source revision and the compiler when CMake configures the build, so it does not see uncommitted changes, is unknown outside a git checkout, and is compared only after the library is opened, when the plugin's static initializers have already run.
The promotion to the global scope lasts for the rest of the process; a module loaded later with its own copy of TENT may bind to the promoted one.
A plugin can only use TENT code the core also contains; a missing symbol is reported when the plugin is loaded, not at build time (the plugin-mode tests catch it).
Plugin mode is exercised by TENT's own tests and the Python guard test; the Store and integration tests assume TCP/RDMA inside TENT and are not run in plugin mode.
The descriptor does not depend on dynamic loading: a registration call that takes a TransportPluginDesc directly would let a statically linked transport use the same interface. It is not part of these PRs; adding it is a small follow-up if packaging needs it.
Packaging (installing the header as an SDK, plugins in the wheel) is left out until the mechanism has been used.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @xiaodouzi666 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4681 ### Changes proposed *Revised after maintainer review: the design below is the reduced one now in #4776 and #4777. The first version of this RFC (ABI fingerprint, reserved enum slots, a name table and wire-format changes, a registry with outcome codes, an SDK, a conformance suite and an inspect tool) is withdrawn; #4778 is closed.* ## Motivation Every TENT transport is compiled into the core. Roadmap #1058 ("Packaging & Usability: enable runtime dynamic loading") asks for a transport to be buildable and shippable apart from the TENT that uses it, so that a device or vendor backend can be packaged on its own, and for a missing or unusable plugin to be diagnosed clearly. ## Design **Interface** (`include/tent/transport_plugin.h`, one header). A plugin is a shared library `libtent_xport_<name>.so` that exports two functions. `tent_transport_plugin_build_id()` is plain C and returns the build ID the plugin was compiled with: the source revision and the compiler, which CMake records in a generated `tent/build_id.h` when the build is configured, plus the libstdc++ string ABI (`_GLIBCXX_USE_CXX11_ABI`). `tent_register_transport_plugin()` (the same word order as `tent_register_device_plugin`) returns a descriptor with the `TransportType` it provides and `create(context)`, which receives the engine's config and topology and returns a `Transport` (or `nullptr` when the plugin does not apply to that engine). A plugin uses TENT's C++ classes directly, so it only runs in the build it was compiled in; the build ID is what TENT checks, through the C function, before it calls any of the plugin's C++ code (as asked in the review of #4776). **Identity.** A plugin provides one of the existing `TransportType` values. The enum, the metadata format, the selector and every transport are unchanged. A new kind of transport still adds its enumerator to TENT, as today. **Loading** (`src/runtime/transport_plugin_loader.cpp`, one call at the end of `loadTransports()`). Nothing is loaded unless `MC_TENT_PLUGIN_DIR` or the `plugins/dir` configuration key names a directory. Files named `libtent_xport_*.so` are opened in name order. A plugin from another build, or one without a build ID, stops the program with a fatal error (`LOG(FATAL)`, like TENT's other configuration errors) that names the plugin and, for a mismatch, both build IDs. Otherwise a plugin only fills a slot the core left empty; a file that cannot be used (it cannot be opened, has no entry point, names an unknown type or a slot already provided, or its `create()` throws or declines) is skipped with one log line; plugins are never unloaded. A plugin links no TENT library and resolves TENT's symbols in the process: an executable that links TENT statically must export them, and a TENT inside an object loaded `RTLD_LOCAL` (a Python extension) is promoted to the global scope before the first plugin is opened. **Built-in transports as plugins** (#4777). `TENT_XPORT_TCP` and `TENT_XPORT_RDMA` choose `static` (the default, unchanged) or `plugin`, in which case TENT does not create the transport and `libtent_xport_tcp.so` / `libtent_xport_rdma.so` are built from the sources of the transport's static library plus a short entry file; they create their transport under the same conditions as the built-in one (`transports/<name>/enable`, and an RDMA NIC in the topology). In plugin mode TENT's tests run against the plugins, including the Python guard test, and a tent-ci leg (`xport-plugin`) builds and tests this mode. ## Size #4776: +918 lines over 12 files (product 309, tests 504, docs 105). #4777: +215/−1 over 9 files, including 6 lines of CI. No existing transport, enum value, metadata field or public API changes. ## Measurements On one 8×H20 host, RDMA `mlx5_4` → `mlx5_2`, 1–8 threads × 4 KiB–64 MiB, read throughput as the median of 3 runs: the plugin build is within 3% of the static build at all 60 points (−2.52% to +0.31%). Below 4 MiB it is up to 2.5% slower, mostly with 1–4 threads; from 4 MiB up the two builds are within 1%, and with two or more threads both reach about 49 GB/s at 8 MiB and above. The read/write mix is consistent at 60/60 points in both builds, and the same plugin-mode binary loads RDMA only when the plugin directory is set. (Measured on an earlier revision of the branches. Since then the loader's checks were reordered and the build ID check was added, and the RDMA plugin changed only in the name of its entry symbol, in how its sources are listed and by the function that returns its build ID.) ## Rollout - PR 1 (#4776): interface, loader, tests, docs. - PR 2 (#4777): TCP and RDMA as plugins, stacked on PR 1. - #4778 (conformance suite, `tent_plugin_inspect`): closed. ## Limitations and possible follow-ups - Same build required: the build ID records the source revision and the compiler when CMake configures the build, so it does not see uncommitted changes, is `unknown` outside a git checkout, and is compared only after the library is opened, when the plugin's static initializers have already run. - The promotion to the global scope lasts for the rest of the process; a module loaded later with its own copy of TENT may bind to the promoted one. - A plugin can only use TENT code the core also contains; a missing symbol is reported when the plugin is loaded, not at build time (the plugin-mode tests catch it). - Plugin mode is exercised by TENT's own tests and the Python guard test; the Store and integration tests assume TCP/RDMA inside TENT and are not run in plugin mode. - The descriptor does not depend on dynamic loading: a registration call that takes a `TransportPluginDesc` directly would let a statically linked transport use the same interface. It is not part of these PRs; adding it is a small follow-up if packaging needs it. - Packaging (installing the header as an SDK, plugins in the wheel) is left out until the mechanism has been used. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:23:48 +00:00

bsakaguchi commented

2026-10-29 21:23:49 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 15, 2026):

Thanks for opening this issue, @xiaodouzi666!

Field	Value
Issue	#4681
GitHub user ID	`77219630`
Reporter	@xiaodouzi666

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 15, 2026): Thanks for opening this issue, @xiaodouzi666! | Field | Value | |-------|-------| | **Issue** | #4681 | | **GitHub user ID** | `77219630` | | **Reporter** | @xiaodouzi666 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:32 +00:00

[PR #1201] [MERGED] [Store]: Add master_bench for benchmarking QPS of MasterService #2068