[GH-ISSUE #109] How does KVCache Client works on 3FS?

bsakaguchi commented

2026-10-26 09:44:23 +00:00

Owner

Originally created by @jinhao2 on GitHub (Mar 6, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/109

The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s.
---- the amazing throughput of KVCache retrieval.
Dose the KVCache storage implemented on 3FS ?
KVCache client should be an kv storage interface, like redis client. Does KVCache Client access storage using 3FS client?
How about the latency KV Cache read ?
Thanks!

Originally created by @jinhao2 on GitHub (Mar 6, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/109 The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s. ---- the amazing throughput of KVCache retrieval. Dose the KVCache storage implemented on 3FS ? KVCache client should be an kv storage interface, like redis client. Does KVCache Client access storage using 3FS client? How about the latency KV Cache read ? Thanks!

bsakaguchi commented

2026-10-26 09:44:24 +00:00

Author

Owner

@kuangdao commented on GitHub (Mar 6, 2025):

m

@kuangdao commented on GitHub (Mar 6, 2025): m

bsakaguchi commented

2026-10-26 09:44:24 +00:00

Author

Owner

@sgua007 commented on GitHub (Mar 6, 2025):

The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s.

Is this 40GiB/s the total peak bandwidth of all client servers reading kvcache? How many client servers are there?

@sgua007 commented on GitHub (Mar 6, 2025): The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s. Is this 40GiB/s the total peak bandwidth of all client servers reading kvcache? How many client servers are there?

bsakaguchi commented

2026-10-26 09:44:25 +00:00

Author

Owner

@echaozh commented on GitHub (Mar 11, 2025):

The peak value is the throughput of 1 client node at any time, but at different times, different nodes may have be the peak.

The KV Cache data is stored on 3FS in two parts, the data part which drives up the throughput, and the meta part which associate the cache data with the tokens. Reading the data using the USRBIO interface is straightforward and very fast. The meta lookup does not require high throughput and may be implemented in different ways. If memory is enough, it can be all in memory.

There's not much black magic here. Just two suggestions: 1. chunk the whole prompt into (big) fixed sized blocks, and 2. prepare all the block reads for a deeper iodepth.

@echaozh commented on GitHub (Mar 11, 2025): The peak value is the throughput of 1 client node at any time, but at different times, different nodes may have be the peak. The KV Cache data is stored on 3FS in two parts, the data part which drives up the throughput, and the meta part which associate the cache data with the tokens. Reading the data using the USRBIO interface is straightforward and very fast. The meta lookup does not require high throughput and may be implemented in different ways. If memory is enough, it can be all in memory. There's not much black magic here. Just two suggestions: 1. chunk the whole prompt into (big) fixed sized blocks, and 2. prepare all the block reads for a deeper iodepth.

bsakaguchi commented

2026-10-26 09:44:25 +00:00

Author

Owner

@wangyibin-gh commented on GitHub (Mar 13, 2025):

3FS-KV should be a middleware built specifically for the inference infrasturcture. Theoretically it is coupled with inference engines like vLLM which schedules prefill/decode instances and knows exactly when and how the kvcache is generated and where it should be transfered (or prefetched). Once a prefetch need is identified, the kvcache index (probably implemented using some in-memory database) is queried and the reader knows where to pull the kvcache chunks. the rest is left to 3FS's client to do USRBIO bulk transfer. One additional thing is that the kvcache index, should be maintained in memory all the time to ensure instant query latency. and it will needs to be kept up-to-date once there's new kvcache or prune ops. last but not least, it should be persisted (in bulk chunk) into 3FS together with the actual kvcache data, periodically. and dont worry about cache miss if the metadata is not updated prmoptly - we can always do calculation upon cache miss. as long as the meta is consistent and there's no false positive match, we are safe. that's why deepseek says - we are doing our best to get a cache hit, but I don't promise on the cache hit rate. :-)

@wangyibin-gh commented on GitHub (Mar 13, 2025): 3FS-KV should be a middleware built specifically for the inference infrasturcture. Theoretically it is coupled with inference engines like vLLM which schedules prefill/decode instances and knows exactly when and how the kvcache is generated and where it should be transfered (or prefetched). Once a prefetch need is identified, the kvcache index (probably implemented using some in-memory database) is queried and the reader knows where to pull the kvcache chunks. the rest is left to 3FS's client to do USRBIO bulk transfer. One additional thing is that the kvcache index, should be maintained in memory all the time to ensure instant query latency. and it will needs to be kept up-to-date once there's new kvcache or prune ops. last but not least, it should be persisted (in bulk chunk) into 3FS together with the actual kvcache data, periodically. and dont worry about cache miss if the metadata is not updated prmoptly - we can always do calculation upon cache miss. as long as the meta is consistent and there's no false positive match, we are safe. that's why deepseek says - we are doing our best to get a cache hit, but I don't promise on the cache hit rate. :-)

bsakaguchi commented

2026-10-26 09:44:25 +00:00

Author

Owner

@gaowayne commented on GitHub (Apr 26, 2025):

@wangyibin-gh @echaozh hello expert, I read through the main page, 3FS KVCache GC IOPS picture, this GC ops, I am guessing this is how vLLM identified KVCache they need compact down to SSD layer, then 3FS take over them, this process called GC ops, instead 40GB/s read BW for each client, this GC actually is write workload from DRAM into 3FS SSD chunk pool? is this correct understanding?

@gaowayne commented on GitHub (Apr 26, 2025): @wangyibin-gh @echaozh hello expert, I read through the main page, 3FS KVCache GC IOPS picture, this GC ops, I am guessing this is how vLLM identified KVCache they need compact down to SSD layer, then 3FS take over them, this process called GC ops, instead 40GB/s read BW for each client, this GC actually is write workload from DRAM into 3FS SSD chunk pool? is this correct understanding?

bsakaguchi commented

2026-10-26 09:44:25 +00:00

Author

Owner

@walsonyang commented on GitHub (May 13, 2025):

The peak value is the throughput of 1 client node at any time, but at different times, different nodes may have be the peak.

The KV Cache data is stored on 3FS in two parts, the data part which drives up the throughput, and the meta part which associate the cache data with the tokens. Reading the data using the USRBIO interface is straightforward and very fast. The meta lookup does not require high throughput and may be implemented in different ways. If memory is enough, it can be all in memory.

There's not much black magic here. Just two suggestions: 1. chunk the whole prompt into (big) fixed sized blocks, and 2. prepare all the block reads for a deeper iodepth.

大神好，这是官方回答吗？一直很困惑deep seek 3FS是如何做kv cache的存储和高性能查询的。
如果是您描述的这种通过“文件系统”存储的方式做的，那它相比于rockdb包个rpc服务的性能优势是什么？好像是为了套文件存储的一种范式，并不是一种最优的范式。毕竟kv cache核心本质是KV，一个KV数据库足矣，不需要“文件系统”那么重。

@walsonyang commented on GitHub (May 13, 2025): > The peak value is the throughput of 1 client node at any time, but at different times, different nodes may have be the peak. > > The KV Cache data is stored on 3FS in two parts, the data part which drives up the throughput, and the meta part which associate the cache data with the tokens. Reading the data using the USRBIO interface is straightforward and very fast. The meta lookup does not require high throughput and may be implemented in different ways. If memory is enough, it can be all in memory. > > There's not much black magic here. Just two suggestions: 1. chunk the whole prompt into (big) fixed sized blocks, and 2. prepare all the block reads for a deeper iodepth. > The peak value is the throughput of 1 client node at any time, but at different times, different nodes may have be the peak. > > The KV Cache data is stored on 3FS in two parts, the data part which drives up the throughput, and the meta part which associate the cache data with the tokens. Reading the data using the USRBIO interface is straightforward and very fast. The meta lookup does not require high throughput and may be implemented in different ways. If memory is enough, it can be all in memory. > > There's not much black magic here. Just two suggestions: 1. chunk the whole prompt into (big) fixed sized blocks, and 2. prepare all the block reads for a deeper iodepth. 大神好，这是官方回答吗？一直很困惑deep seek 3FS是如何做kv cache的存储和高性能查询的。如果是您描述的这种通过“文件系统”存储的方式做的，那它相比于rockdb包个rpc服务的性能优势是什么？好像是为了套文件存储的一种范式，并不是一种最优的范式。毕竟kv cache核心本质是KV，一个KV数据库足矣，不需要“文件系统”那么重。

bsakaguchi commented

2026-10-26 09:44:26 +00:00

Author

Owner

@gaowayne commented on GitHub (Jun 22, 2025):

@walsonyang expert

if deepseek 3FS currently support latest version vllm kv_transfer_config SharedStorageConnector?
if you can share detail step how to run through this with single node one GPU to make save kv happening?

@gaowayne commented on GitHub (Jun 22, 2025): @walsonyang expert if deepseek 3FS currently support latest version vllm kv_transfer_config SharedStorageConnector? if you can share detail step how to run through this with single node one GPU to make save kv happening?

bsakaguchi referenced this issue

2026-10-26 09:48:09 +00:00

[PR #61] [CLOSED] Refactor: Replace magic number with constexpr for better readability #274

Rows
Columns

[GH-ISSUE #109] How does KVCache Client works on 3FS? #61