[GH-ISSUE #4676] [Bug]: [Misc] SSD admin CLI tools rebuild full SSH handshake per remote command #1190

New issue

Open

opened 2026-10-29 21:23:46 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:46 +00:00

Owner

Originally created by @thrower01 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4676

Bug Report

ontext

Bothpython/mooncake/mooncake_ssd_register.py::_execute_ssh_command andmooncake_ssd_unregister.py::_execute_ssh_command open a freshparamiko.SSHClient , runconnect() , run one command, then callclose() . No connection reuse.

The register tool calls this function for every SPDK target, and within one target the pattern is:

probescripts/rpc.py location (up to 2test -f commands)
callnvmf_get_subsystems
for each namespace, callbdev_get_bdevs
That is 3 to 5 full SSH handshakes per target — TCP SYN → KEX → userauth → channel open → exec → channel close. Each target pays the handshake multiplier once per discovery run. The unregister tool has the same duplicated code path.

Fix direction

Introduce a process-wideSSHConnectionPool inpython/mooncake/_administration.py keyed by(ip, port, username, password, key_file) :

lazy connect on first_get_client(key)
get_transport().is_active() health check before reuse
single automatic reconnect + retry onexec_command failure (handles transient TCP drops)
close_all() /reset_for_test() for cleanup
Both admin scripts delegate_execute_ssh_command() to the pool. Each SPDK target now shares one SSH handshake across all commands in its discovery flow.

Impact

Reduces per-target handshake count fromO(namespaces + 3) toO(1) ; eliminates repeated KEX CPU work on the remote node; and makes the discovery path resilient to a single transient SSH drop per target. Only affects the two admin CLI tools, which are operator-facing and off the inference / training data path.

Acceptance

New unit tests covering same-host reuse (1 connect for N commands), cross-host isolation (one connect per distinct host), and reconnect-once after a transient failure.
Existingrequire_paramiko lazy-import tests keep passing

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @thrower01 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4676 ### Bug Report ontext Both`python/mooncake/mooncake_ssd_register.py::_execute_ssh_command` and`mooncake_ssd_unregister.py::_execute_ssh_command` open a fresh`paramiko.SSHClient` , run`connect()` , run one command, then call`close()` . No connection reuse. The register tool calls this function for every SPDK target, and within one target the pattern is: 1. probe`scripts/rpc.py` location (up to 2`test -f` commands) 2. call`nvmf_get_subsystems` 3. for each namespace, call`bdev_get_bdevs` That is 3 to 5 full SSH handshakes per target — TCP SYN → KEX → userauth → channel open → exec → channel close. Each target pays the handshake multiplier once per discovery run. The unregister tool has the same duplicated code path. Fix direction Introduce a process-wide`SSHConnectionPool` in`python/mooncake/_administration.py` keyed by`(ip, port, username, password, key_file)` : - lazy connect on first`_get_client(key)` - `get_transport().is_active()` health check before reuse - single automatic reconnect + retry on`exec_command` failure (handles transient TCP drops) - `close_all()` /`reset_for_test()` for cleanup Both admin scripts delegate`_execute_ssh_command()` to the pool. Each SPDK target now shares one SSH handshake across all commands in its discovery flow. Impact Reduces per-target handshake count from`O(namespaces + 3)` to`O(1)` ; eliminates repeated KEX CPU work on the remote node; and makes the discovery path resilient to a single transient SSH drop per target. Only affects the two admin CLI tools, which are operator-facing and off the inference / training data path. Acceptance - New unit tests covering same-host reuse (1 connect for N commands), cross-host isolation (one connect per distinct host), and reconnect-once after a transient failure. - Existing`require_paramiko` lazy-import tests keep passing ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:46 +00:00

bsakaguchi commented

2026-10-29 21:23:47 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 15, 2026):

Thanks for opening this issue, @thrower01!

Field	Value
Issue	#4676
GitHub user ID	`340463471`
Reporter	@thrower01

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 15, 2026): Thanks for opening this issue, @thrower01! | Field | Value | |-------|-------| | **Issue** | #4676 | | **GitHub user ID** | `340463471` | | **Reporter** | @thrower01 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:29 +00:00

[PR #1190] [MERGED] [yalantinglibs]set ylt log level with env #2061