[GH-ISSUE #5001] [Feature Request]: Add a Linux kernel NVMe-oF initiator backend #1269

New issue

Open

opened 2026-10-29 21:24:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:39 +00:00

Owner

Originally created by @cage-goat on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5001

PR #3424 decoupled Mooncake Store’s NoF data path from SPDK, providing a foundation for additional initiator backends. Building on this work, we have implemented a Linux kernel NVMe-oF initiator backend for Mooncake Store.

The existing SPDK initiator has several related limitations:

#3863: Clients continue reusing failed qpairs and cannot resume I/O after the target recovers.
#4768: Client connections lack KATO, allowing stale controllers and qpairs to remain on the target after connection failures.
#4805: Master heartbeat connections cannot recover after the target becomes available again.
#4807: Master heartbeat connections also lack KATO, leaving stale connection resources on the target.
#3864: NoF operations may wait indefinitely when I/O neither completes nor returns an error.
#3795: Mooncake lacks stable, configurable Host NQN support and relies on SPDK’s default identity, which may be rejected by targets enforcing Host NQN access control.

The Linux NVMe-oF initiator already provides controller reconnection, queue recovery, Keep Alive, and I/O timeout handling. The new backend allows both client data I/O and Master heartbeat probes to reuse these mechanisms, reducing the connection lifecycle and recovery logic maintained within Mooncake. Recovery behavior and I/O waiting times depend on the kernel and the timeout policies configured by deployment tooling.

The Linux backend also reuses the Host NQN configuration available through the operating system and nvme-cli, making it easier to work with targets enforcing host access control. It provides an alternative to the existing SPDK backend without requiring SPDK/DPDK or hugepages on initiator hosts.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @cage-goat on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5001 [PR #3424](https://github.com/kvcache-ai/Mooncake/pull/3424) decoupled Mooncake Store’s NoF data path from SPDK, providing a foundation for additional initiator backends. Building on this work, we have implemented a Linux kernel NVMe-oF initiator backend for Mooncake Store. The existing SPDK initiator has several related limitations: - [#3863](https://github.com/kvcache-ai/Mooncake/issues/3863): Clients continue reusing failed qpairs and cannot resume I/O after the target recovers. - [#4768](https://github.com/kvcache-ai/Mooncake/issues/4768): Client connections lack KATO, allowing stale controllers and qpairs to remain on the target after connection failures. - [#4805](https://github.com/kvcache-ai/Mooncake/issues/4805): Master heartbeat connections cannot recover after the target becomes available again. - [#4807](https://github.com/kvcache-ai/Mooncake/issues/4807): Master heartbeat connections also lack KATO, leaving stale connection resources on the target. - [#3864](https://github.com/kvcache-ai/Mooncake/issues/3864): NoF operations may wait indefinitely when I/O neither completes nor returns an error. - [#3795](https://github.com/kvcache-ai/Mooncake/issues/3795): Mooncake lacks stable, configurable Host NQN support and relies on SPDK’s default identity, which may be rejected by targets enforcing Host NQN access control. The Linux NVMe-oF initiator already provides controller reconnection, queue recovery, Keep Alive, and I/O timeout handling. The new backend allows both client data I/O and Master heartbeat probes to reuse these mechanisms, reducing the connection lifecycle and recovery logic maintained within Mooncake. Recovery behavior and I/O waiting times depend on the kernel and the timeout policies configured by deployment tooling. The Linux backend also reuses the Host NQN configuration available through the operating system and `nvme-cli`, making it easier to work with targets enforcing host access control. It provides an alternative to the existing SPDK backend without requiring SPDK/DPDK or hugepages on initiator hosts. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi commented

2026-10-29 21:24:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 27, 2026):

Thanks for opening this issue, @cage-goat!

Field	Value
Issue	#5001
GitHub user ID	`318081269`
Reporter	@cage-goat

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 27, 2026): Thanks for opening this issue, @cage-goat! | Field | Value | |-------|-------| | **Issue** | #5001 | | **GitHub user ID** | `318081269` | | **Reporter** | @cage-goat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:57 +00:00

[PR #1269] [CLOSED] [EP] Avoid server name collision during EP backend init #2121