[GH-ISSUE #4996] [RFC]: Valkey as an object storage offload target, with RDMA through Transfer Engine #1268

New issue

Open

opened 2026-10-29 21:24:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:39 +00:00

Owner

Originally created by @whn09 on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4996

Changes proposed

Motivation

Store offload can write objects to OSS or an S3-compatible service (ObjectStorageAdapter under DistributedStorageBackend). Both need a bucket and move every byte over HTTP. Many deployments already run Valkey as an in-VPC cache tier. With the valkey-large-object module (BLOB.* commands), Valkey stores multi-MB binary objects in DRAM or tiered to NVMe, which fits KV-cache objects.

An adapter for it gives Store a remote, shared offload tier with no bucket and no filesystem mount. The module can also move object bytes with one-sided RDMA. If its server reaches the owner's buffers through Transfer Engine, offload and load skip the RESP connection and the copies on both ends. Across two r8idn.96xlarge hosts (2 EFA NICs, 600 Gbps), this runs at 57 GB/s for offload and 63 GB/s for load, against 3.4 and 1.3 GB/s over TCP.

Proposed design

Adapter. ValkeyObjectStorageAdapter is a third ObjectStorageAdapter, selected with MOONCAKE_DISTRIBUTED_FS_TYPE=valkey.

Each logical key is one large object under <prefix>/<logical key>. The prefix is MOONCAKE_DISTRIBUTED_ROOT_DIR without leading and trailing slashes, as the OSS adapter does. Keys are binary-safe.
Put/PutV/Get use BLOB.TCP_SET and BLOB.TCP_GET, Exists uses EXISTS, Delete uses DEL. ListKeys uses SCAN with an escaped MATCH pattern, and gets sizes from pipelined BLOB.INFO.
A fixed pool of hiredis connections (MOONCAKE_VALKEY_POOL_SIZE, default 8). Batch requests are spread across the pool and pipelined per connection.
A connection that hits an I/O error is replaced on release. AUTH is sent on every new connection.
Init() fails fast if the server lacks BLOB.TCP_SET.
Built under the existing STORE_USE_REDIS option, which already links hiredis. No new dependency.

RDMA path (MOONCAKE_VALKEY_TRANSPORT=rdma).

New virtual ObjectStorageAdapter::BindTransferEndpoint(segment_name), with a no-op default. DistributedStorageBackend forwards it. In object storage mode, FileStorage binds client_->GetSegmentEndpoint(). Other adapters are unchanged.
Each pooled connection sends BLOB.RDMA_HELLO <segment> once. The server then opens the owner's segment with its own Transfer Engine.
Offload: BLOB.RDMA_SET key total_len (rkey addr len).... The server pulls the PutV iovecs straight out of the owner's mounted segment, so there is no joining copy.
Load: BLOB.RDMA_GET key (rkey addr len), pipelined per connection. The server pushes the object into FileStorage's registered client buffer.
The owner publishes no new metadata. The server addresses the owner's buffers by segment name and virtual address, which Transfer Engine resolves; the rkey field is unused.

Fallback. Correctness never depends on registration.

A request the server cannot carry over RDMA is redone over TCP: an unregistered buffer (for example a GPU staging buffer), a read shorter than the object, or a failed HELLO.
Errors that BLOB.TCP_SET would return as well are not resent: OOM, WRONGTYPE, max object size, zero length.
GetTransportStats() counts RDMA requests, TCP requests and fallbacks.

Configuration. MOONCAKE_VALKEY_{ADDR,USERNAME,PASSWORD,POOL_SIZE,TIMEOUT_MS,TRANSPORT}. The default is TCP, so nothing changes for deployments that do not set MOONCAKE_DISTRIBUTED_FS_TYPE=valkey.

Dependencies

The RDMA path needs the module's Transfer Engine backend on the server (fabric-provider MooncakeTe). It is prototyped in whn09/valkey-large-object@mooncake-te-backend and is not in a module release yet. The TCP path works with released modules.
The server registers its pool with remote_accessible=false, so peers cannot target it. On EFA this relies on #4976, which makes the EFA transport honor that flag.

Trust model

With RDMA, the Valkey server reads and writes the owner's registered memory, the mounted segment and the client buffers. This is the same trust any Transfer Engine peer that can open the segment already has. Run the server inside the same trust boundary as the Store clients. TCP mode exposes no memory.

Limitations

The module rejects empty objects and objects over its max-object-size (512 MiB by default), so offloading those fails.
Valkey evicts keys under its own maxmemory-policy, and an evicted object reads as missing. Use noeviction, or size the server for the offloaded data, when offloaded objects must stay readable.
On multi-NIC hosts Transfer Engine sends a host buffer over its NUMA node's NICs. Spreading the server pool across nodes (the module's te-locations) is what reaches every NIC.

Out of scope

Valkey Cluster (key slots across shards). The adapter talks to one server.
TLS on the RESP connection.
Using Valkey as a primary Store tier rather than an offload target.

Validation (2× r8idn.96xlarge, 2 EFA NICs each, libfabric 2.4, hiredis 1.2.0)

Store runs on one host; Valkey with the module and its TE backend (te-locations cpu:0,cpu:3) runs on the other, so all traffic crosses hosts.

valkey_adapter_live_test, 14 cases:
- 10 over TCP: round trip, multi-region PutV, prefix reads, missing keys, overwrite, binary keys with \0 and glob characters, Exists/Delete/ListKeys, empty objects, batches, health check, and DistributedStorageBackend offload/exists/load/scan.
- 4 over RDMA: multi-region put/get and a 32-object batch go entirely over RDMA with 0 fallbacks. An unregistered heap buffer and a short prefix read fall back to TCP and succeed.
FileStorageTest end to end with a real master and client:
- Over TCP, the master reports a LOCAL_DISK replica per object and the bytes match.
- Over RDMA (efa, P2PHANDSHAKE, 16 × 4 MiB), the server's largeobj_rdma_read_bytes and largeobj_rdma_write_bytes each grow by exactly the payload size.
Throughput, PutBatch/GetBatch over 4 GiB per round:

Object Pool TCP put TCP get RDMA put RDMA get

1 MiB 32 3.7 GB/s 1.3 GB/s 58.2 GB/s 62.7 GB/s

4 MiB 32 3.4 GB/s 1.3 GB/s 57.3 GB/s 63.1 GB/s

64 MiB 64 — — 56.7 GB/s 64.2 GB/s

RDMA reaches about 85% of line rate. TCP GET is bound by the Valkey server: its main thread and I/O threads sit at 100% CPU.

Object	Pool	TCP put	TCP get	RDMA put	RDMA get
1 MiB	32	3.7 GB/s	1.3 GB/s	58.2 GB/s	62.7 GB/s
4 MiB	32	3.4 GB/s	1.3 GB/s	57.3 GB/s	63.1 GB/s
64 MiB	64	—	—	56.7 GB/s	64.2 GB/s

Implementation: #4991.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @whn09 on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4996 ### Changes proposed ## Motivation Store offload can write objects to OSS or an S3-compatible service (`ObjectStorageAdapter` under `DistributedStorageBackend`). Both need a bucket and move every byte over HTTP. Many deployments already run Valkey as an in-VPC cache tier. With the [valkey-large-object](https://github.com/valkey-io/valkey-large-object) module (`BLOB.*` commands), Valkey stores multi-MB binary objects in DRAM or tiered to NVMe, which fits KV-cache objects. An adapter for it gives Store a remote, shared offload tier with no bucket and no filesystem mount. The module can also move object bytes with one-sided RDMA. If its server reaches the owner's buffers through Transfer Engine, offload and load skip the RESP connection and the copies on both ends. Across two r8idn.96xlarge hosts (2 EFA NICs, 600 Gbps), this runs at 57 GB/s for offload and 63 GB/s for load, against 3.4 and 1.3 GB/s over TCP. ## Proposed design **Adapter.** `ValkeyObjectStorageAdapter` is a third `ObjectStorageAdapter`, selected with `MOONCAKE_DISTRIBUTED_FS_TYPE=valkey`. - Each logical key is one large object under `<prefix>/<logical key>`. The prefix is `MOONCAKE_DISTRIBUTED_ROOT_DIR` without leading and trailing slashes, as the OSS adapter does. Keys are binary-safe. - `Put`/`PutV`/`Get` use `BLOB.TCP_SET` and `BLOB.TCP_GET`, `Exists` uses `EXISTS`, `Delete` uses `DEL`. `ListKeys` uses `SCAN` with an escaped `MATCH` pattern, and gets sizes from pipelined `BLOB.INFO`. - A fixed pool of hiredis connections (`MOONCAKE_VALKEY_POOL_SIZE`, default 8). Batch requests are spread across the pool and pipelined per connection. - A connection that hits an I/O error is replaced on release. `AUTH` is sent on every new connection. - `Init()` fails fast if the server lacks `BLOB.TCP_SET`. - Built under the existing `STORE_USE_REDIS` option, which already links hiredis. No new dependency. **RDMA path** (`MOONCAKE_VALKEY_TRANSPORT=rdma`). - New virtual `ObjectStorageAdapter::BindTransferEndpoint(segment_name)`, with a no-op default. `DistributedStorageBackend` forwards it. In object storage mode, `FileStorage` binds `client_->GetSegmentEndpoint()`. Other adapters are unchanged. - Each pooled connection sends `BLOB.RDMA_HELLO <segment>` once. The server then opens the owner's segment with its own Transfer Engine. - Offload: `BLOB.RDMA_SET key total_len (rkey addr len)...`. The server pulls the `PutV` iovecs straight out of the owner's mounted segment, so there is no joining copy. - Load: `BLOB.RDMA_GET key (rkey addr len)`, pipelined per connection. The server pushes the object into FileStorage's registered client buffer. - The owner publishes no new metadata. The server addresses the owner's buffers by segment name and virtual address, which Transfer Engine resolves; the rkey field is unused. **Fallback.** Correctness never depends on registration. - A request the server cannot carry over RDMA is redone over TCP: an unregistered buffer (for example a GPU staging buffer), a read shorter than the object, or a failed HELLO. - Errors that `BLOB.TCP_SET` would return as well are not resent: OOM, `WRONGTYPE`, max object size, zero length. - `GetTransportStats()` counts RDMA requests, TCP requests and fallbacks. **Configuration.** `MOONCAKE_VALKEY_{ADDR,USERNAME,PASSWORD,POOL_SIZE,TIMEOUT_MS,TRANSPORT}`. The default is TCP, so nothing changes for deployments that do not set `MOONCAKE_DISTRIBUTED_FS_TYPE=valkey`. ## Dependencies - The RDMA path needs the module's Transfer Engine backend on the server (`fabric-provider MooncakeTe`). It is prototyped in [whn09/valkey-large-object@mooncake-te-backend](https://github.com/whn09/valkey-large-object/tree/mooncake-te-backend) and is not in a module release yet. The TCP path works with released modules. - The server registers its pool with `remote_accessible=false`, so peers cannot target it. On EFA this relies on #4976, which makes the EFA transport honor that flag. ## Trust model With RDMA, the Valkey server reads and writes the owner's registered memory, the mounted segment and the client buffers. This is the same trust any Transfer Engine peer that can open the segment already has. Run the server inside the same trust boundary as the Store clients. TCP mode exposes no memory. ## Limitations - The module rejects empty objects and objects over its `max-object-size` (512 MiB by default), so offloading those fails. - Valkey evicts keys under its own `maxmemory-policy`, and an evicted object reads as missing. Use `noeviction`, or size the server for the offloaded data, when offloaded objects must stay readable. - On multi-NIC hosts Transfer Engine sends a host buffer over its NUMA node's NICs. Spreading the server pool across nodes (the module's `te-locations`) is what reaches every NIC. ## Out of scope - Valkey Cluster (key slots across shards). The adapter talks to one server. - TLS on the RESP connection. - Using Valkey as a primary Store tier rather than an offload target. ## Validation (2× r8idn.96xlarge, 2 EFA NICs each, libfabric 2.4, hiredis 1.2.0) Store runs on one host; Valkey with the module and its TE backend (`te-locations cpu:0,cpu:3`) runs on the other, so all traffic crosses hosts. - `valkey_adapter_live_test`, 14 cases: - 10 over TCP: round trip, multi-region `PutV`, prefix reads, missing keys, overwrite, binary keys with `\0` and glob characters, `Exists`/`Delete`/`ListKeys`, empty objects, batches, health check, and `DistributedStorageBackend` offload/exists/load/scan. - 4 over RDMA: multi-region put/get and a 32-object batch go entirely over RDMA with 0 fallbacks. An unregistered heap buffer and a short prefix read fall back to TCP and succeed. - `FileStorageTest` end to end with a real master and client: - Over TCP, the master reports a `LOCAL_DISK` replica per object and the bytes match. - Over RDMA (`efa`, `P2PHANDSHAKE`, 16 × 4 MiB), the server's `largeobj_rdma_read_bytes` and `largeobj_rdma_write_bytes` each grow by exactly the payload size. - Throughput, `PutBatch`/`GetBatch` over 4 GiB per round: | Object | Pool | TCP put | TCP get | RDMA put | RDMA get | |---|---|---|---|---|---| | 1 MiB | 32 | 3.7 GB/s | 1.3 GB/s | 58.2 GB/s | 62.7 GB/s | | 4 MiB | 32 | 3.4 GB/s | 1.3 GB/s | 57.3 GB/s | 63.1 GB/s | | 64 MiB | 64 | — | — | 56.7 GB/s | 64.2 GB/s | RDMA reaches about 85% of line rate. TCP GET is bound by the Valkey server: its main thread and I/O threads sit at 100% CPU. Implementation: #4991. ### Before submitting a new issue... - [X] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:24:39 +00:00

bsakaguchi commented

2026-10-29 21:24:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 27, 2026):

Thanks for opening this issue, @whn09!

Field	Value
Issue	#4996
GitHub user ID	`964527`
Reporter	@whn09

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 27, 2026): Thanks for opening this issue, @whn09! | Field | Value | |-------|-------| | **Issue** | #4996 | | **GitHub user ID** | `964527` | | **Reporter** | @whn09 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:55 +00:00

[PR #1268] [MERGED] [Store] feat Uncouple client segment meta from allocator #2119