[GH-ISSUE #467] mooncake X LMCache performance

bsakaguchi commented

2026-10-29 21:10:20 +00:00

Owner

Originally created by @cicirori on GitHub (Jun 10, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/467

hi, this is the data I've collected on LMCache's Mooncake backends. The test is a very direct and focused benchmark aimed at validating the end-to-end performance of these backends when integrated with LMCache and vLLM:

GPU memory usage is restricted to ensure vLLM’s own GPU prompt caching is not triggered.
32 requests of 10k tokens each are issued concurrently, with max_new_tokens set to 1—so the end-to-end performance essentially reflects the TTFT, which is depended by the backend's store/retrieve performance and implementation efficiency.

Mooncake turned out to be unexpectedly slow:

	first run(kv cache store)	second run(kv cache retrieve)
baseline	8.46s	8.47s
mooncake	12.10s	29.00s
gds	14.04s	2.70s
local dram	9.42s	1.77s

After investigation, I’ve identified a few potential causes.

There's a noticeable overhead in each store/retrieve operation when using mooncake store with LMCache. I haven’t had the chance to pinpoint the exact line of code causing it yet.
Mooncake store uses 4MB page sizes for memory management internally. When handling reads larger than 4MB, it incurs highly inefficient memory allocation and copying, which can take several times longer than the actual data transfer. If we limit each read to under 4MB to avoid this issue, it significantly increases the overhead mentioned in point 1.

I’d like to know if this is expected behavior or if I’m using it incorrectly.

Originally created by @cicirori on GitHub (Jun 10, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/467 hi, this is the data I've collected on LMCache's Mooncake backends. The test is a very direct and focused benchmark aimed at validating the end-to-end performance of these backends when integrated with LMCache and vLLM: 1. GPU memory usage is restricted to ensure vLLM’s own GPU prompt caching is not triggered. 2. 32 requests of 10k tokens each are issued concurrently, with `max_new_tokens` set to 1—so the end-to-end performance essentially reflects the TTFT, which is depended by the backend's store/retrieve performance and implementation efficiency. Mooncake turned out to be unexpectedly slow: | first run(kv cache store) | second run(kv cache retrieve) -- | -- | -- baseline | 8.46s | 8.47s mooncake | 12.10s | 29.00s gds | 14.04s | 2.70s local dram | 9.42s | 1.77s After investigation, I’ve identified a few potential causes. 1. There's a noticeable overhead in each store/retrieve operation when using mooncake store with LMCache. I haven’t had the chance to pinpoint the exact line of code causing it yet. 2. Mooncake store uses 4MB page sizes for memory management internally. When handling reads larger than 4MB, it incurs highly inefficient memory allocation and copying, which can take several times longer than the actual data transfer. If we limit each read to under 4MB to avoid this issue, it significantly increases the overhead mentioned in point 1. I’d like to know if this is expected behavior or if I’m using it incorrectly.

bsakaguchi

2026-10-29 21:10:20 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:21 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 10, 2025):

I'm running a llama 3.1 8B with 1 gpu,

LMCache config:

    env_vars = {
        "LMCACHE_USE_EXPERIMENTAL": "True",
        "LMCACHE_TRACK_USAGE": "False",
        "LMCACHE_CHUNK_SIZE": "16",
        "LMCACHE_LOCAL_CPU": "False",
        "LMCACHE_MAX_LOCAL_CPU_SIZE": "10.0",
        "LMCACHE_REMOTE_URL": "mooncakestore://ip:50051/",
        "LMCACHE_REMOTE_SERDE": "naive",
        "LMCACHE_PIPELINED_BACKEND": "False",
        "MOONCAKE_CONFIG_PATH": "/path/to/mooncake.json",
    }

mooncache config:

{
    "local_hostname": "ip",
    "metadata_server": "P2PHANDSHAKE",
    "protocol": "rdma",
    "device_name": "mlx5_0",
    "master_server_address": "ip:50051",
    "local_buffer_size": 10737418240,
    "global_segment_size": 338228674560
}

@cicirori commented on GitHub (Jun 10, 2025): I'm running a llama 3.1 8B with 1 gpu, LMCache config: ``` env_vars = { "LMCACHE_USE_EXPERIMENTAL": "True", "LMCACHE_TRACK_USAGE": "False", "LMCACHE_CHUNK_SIZE": "16", "LMCACHE_LOCAL_CPU": "False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "10.0", "LMCACHE_REMOTE_URL": "mooncakestore://ip:50051/", "LMCACHE_REMOTE_SERDE": "naive", "LMCACHE_PIPELINED_BACKEND": "False", "MOONCAKE_CONFIG_PATH": "/path/to/mooncake.json", } ``` mooncache config: ``` { "local_hostname": "ip", "metadata_server": "P2PHANDSHAKE", "protocol": "rdma", "device_name": "mlx5_0", "master_server_address": "ip:50051", "local_buffer_size": 10737418240, "global_segment_size": 338228674560 } ```

bsakaguchi commented

2026-10-29 21:10:21 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 11, 2025):

Could you try increasing LMCACHE_CHUNK_SIZE from 16 to 256? The current value is a bit too small for our case. We're not using 4MB pages, and based on our testing, a chunk size around 15MB works much better.

You can check the metrics on the master node to confirm this. Also, please make sure Mooncake is working correctly—LMCache logs should help with that.

@xiaguan commented on GitHub (Jun 11, 2025): Could you try increasing LMCACHE_CHUNK_SIZE from 16 to 256? The current value is a bit too small for our case. We're not using 4MB pages, and based on our testing, a chunk size around 15MB works much better. You can check the metrics on the master node to confirm this. Also, please make sure Mooncake is working correctly—LMCache logs should help with that.

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 11, 2025):

@xiaguan Hi, thanks for your reply.

Ah, I missed that the test with chunksize 256 was done earlier. The results are slightly better, but not significantly.

	first run(kv cache store)	second run(kv cache retrieve)
baseline	8.46s	8.47s
mooncake(chunk size 256)	12.10s	29.00s
gds	14.04s	2.70s
local dram	9.42s	1.77s
mooncake(chunk size 16)	34.13s	48.72s

@cicirori commented on GitHub (Jun 11, 2025): @xiaguan Hi, thanks for your reply. Ah, I missed that the test with chunksize 256 was done earlier. The results are slightly better, but not significantly. | first run(kv cache store) | second run(kv cache retrieve) -- | -- | -- baseline | 8.46s | 8.47s mooncake(chunk size 256) | 12.10s | 29.00s gds | 14.04s | 2.70s local dram | 9.42s | 1.77s mooncake(chunk size 16) | 34.13s | 48.72s

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 11, 2025):

I used https://lmcache.ai/kv_cache_calculator.html to estimate that a chunk size of 256 results in approximately 32MB, while reducing it to 128 keeps it around 15MB, which seems to be the optimal value.

Moreover, you can enable the logs for this connector to observe the actual data size being transferred and the time consumed. This will allow you to calculate the bandwidth of mooncake, which theoretically should be close to the RDMA network card's bandwidth: github.com/LMCache/LMCache@2a521e3325/lmcache/v1/storage_backend/connector/instrumented_connector.py.

BTW, the performance metrics in the report are based on the use of the etcd as the transfer engine's metdata server. However, the current release does not include the etcd plugin. You need compile it with -DUSE_ETCD. It will be bundled in the upcoming release.

@xiaguan commented on GitHub (Jun 11, 2025): I used https://lmcache.ai/kv_cache_calculator.html to estimate that a chunk size of 256 results in approximately 32MB, while reducing it to 128 keeps it around 15MB, which seems to be the optimal value. Moreover, you can enable the logs for this connector to observe the actual data size being transferred and the time consumed. This will allow you to calculate the bandwidth of mooncake, which theoretically should be close to the RDMA network card's bandwidth: https://github.com/LMCache/LMCache/blob/2a521e33252457b9dff5708e7692643f80f59f82/lmcache/v1/storage_backend/connector/instrumented_connector.py. BTW, the performance metrics in the report are based on the use of the etcd as the transfer engine's metdata server. However, the current release does not include the etcd plugin. You need compile it with `-DUSE_ETCD`. It will be bundled in the upcoming release.

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 11, 2025):

@xiaguan Thanks for your quick reply again!

After switching to etcd with a chunk size of 128, there was some improvement, but not significant.

	first run(kv cache store)	second run(kv cache retrieve)
baseline	8.46s	8.47s
mooncake(chunk size 256)	12.10s	29.00s
gds	14.04s	2.70s
local dram	9.42s	1.77s
mooncake(chunk size 16)	34.13s	48.72s
mooncake(chunk size 128 + etcd)	11.94s	27.67s

Below is the debug output from the instrument connector:

// store
[1m[2025-06-11 13:00:09,409] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 6.168234 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:09,409] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.648038 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:09,410] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.496377 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:09,412] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.618691 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:09,414] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.926752 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m

// retrieve
[1m[2025-06-11 13:00:43,770] LMCache DEBUG:[0m Get takes 9.439260 msec, deserialization takes 0.005337 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m
[1m[2025-06-11 13:00:43,780] LMCache DEBUG:[0m Bytes loaded: 16.7772 MBytes, and takes 9.053315 msec [3m(instrumented_connector.py:61:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:43,780] LMCache DEBUG:[0m Get takes 9.390520 msec, deserialization takes 0.004178 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m
[1m[2025-06-11 13:00:43,790] LMCache DEBUG:[0m Bytes loaded: 16.7772 MBytes, and takes 9.200347 msec [3m(instrumented_connector.py:61:lmcache.v1.storage_backend.connector.instrumented_connector)[0m
[1m[2025-06-11 13:00:43,790] LMCache DEBUG:[0m Get takes 9.546404 msec, deserialization takes 0.004401 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m

The bandwidth doesn’t look ideal. In addition to these overheads, I noticed from the timeline that there are quite a few copies between CPUs and from CPU to GPU. These overheads combined are likely non-trivial as well.

etcd and mooncake_master launch cmd:

bin/etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
mooncake_master -v=1 -port=50051 -max_threads 64 -metrics_port 9004

@cicirori commented on GitHub (Jun 11, 2025): @xiaguan Thanks for your quick reply again! After switching to etcd with a chunk size of 128, there was some improvement, but not significant. | first run(kv cache store) | second run(kv cache retrieve) -- | -- | -- baseline | 8.46s | 8.47s mooncake(chunk size 256) | 12.10s | 29.00s gds | 14.04s | 2.70s local dram | 9.42s | 1.77s mooncake(chunk size 16) | 34.13s | 48.72s mooncake(chunk size 128 + etcd) | 11.94s | 27.67s Below is the debug output from the instrument connector: ``` // store [1m[2025-06-11 13:00:09,409] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 6.168234 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:09,409] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.648038 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:09,410] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.496377 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:09,412] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.618691 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:09,414] LMCache DEBUG:[0m Bytes offloaded: 16.7772 MBytes, and takes 5.926752 msec [3m(instrumented_connector.py:51:lmcache.v1.storage_backend.connector.instrumented_connector)[0m // retrieve [1m[2025-06-11 13:00:43,770] LMCache DEBUG:[0m Get takes 9.439260 msec, deserialization takes 0.005337 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m [1m[2025-06-11 13:00:43,780] LMCache DEBUG:[0m Bytes loaded: 16.7772 MBytes, and takes 9.053315 msec [3m(instrumented_connector.py:61:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:43,780] LMCache DEBUG:[0m Get takes 9.390520 msec, deserialization takes 0.004178 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m [1m[2025-06-11 13:00:43,790] LMCache DEBUG:[0m Bytes loaded: 16.7772 MBytes, and takes 9.200347 msec [3m(instrumented_connector.py:61:lmcache.v1.storage_backend.connector.instrumented_connector)[0m [1m[2025-06-11 13:00:43,790] LMCache DEBUG:[0m Get takes 9.546404 msec, deserialization takes 0.004401 msec [3m(remote_backend.py:219:lmcache.v1.storage_backend.remote_backend)[0m ``` The bandwidth doesn’t look ideal. In addition to these overheads, I noticed from the timeline that there are quite a few copies between CPUs and from CPU to GPU. These overheads combined are likely non-trivial as well. etcd and mooncake_master launch cmd: ``` bin/etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 mooncake_master -v=1 -port=50051 -max_threads 64 -metrics_port 9004 ```

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 11, 2025):

The latency looks incorrect. Is the lmcache on a separate machine from the master, causing latency, or are they on the same machine? If on the same machine, RPC should be very fast (tens of microseconds). The bandwidth seems abnormal for an RDMA NIC; a 100Gbps card should complete operations within 2ms. The root cause is unclear. Could you test the RDMA NIC bandwidth or use the transfer engine bench? https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/example/transfer_engine_bench.cpp

@xiaguan commented on GitHub (Jun 11, 2025): The latency looks incorrect. Is the lmcache on a separate machine from the master, causing latency, or are they on the same machine? If on the same machine, RPC should be very fast (tens of microseconds). The bandwidth seems abnormal for an RDMA NIC; a 100Gbps card should complete operations within 2ms. The root cause is unclear. Could you test the RDMA NIC bandwidth or use the transfer engine bench? https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/example/transfer_engine_bench.cpp

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 11, 2025):

yeah, they are on the same machine, here is the transfer engine bench result with default config.

Test completed: duration 10.00, batch count 249891, throughput 13.10 GB/s (read)
Test completed: duration 10.00, batch count 268314, throughput 14.07 GB/s (write)

didn't know if it is as excepted, not reached the nic speed limit, but much more better than instrument connector's outputs.

update: after adjusted(changed block size to 32k):

Test completed: duration 10.00, batch count 58254, throughput 48.86 GB/s (read)
Test completed: duration 10.00, batch count 60657, throughput 50.88 GB/s (write)

@cicirori commented on GitHub (Jun 11, 2025): yeah, they are on the same machine, here is the transfer engine bench result with default config. ``` Test completed: duration 10.00, batch count 249891, throughput 13.10 GB/s (read) Test completed: duration 10.00, batch count 268314, throughput 14.07 GB/s (write) ``` didn't know if it is as excepted, not reached the nic speed limit, but much more better than instrument connector's outputs. update: after adjusted(changed block size to 32k): ``` Test completed: duration 10.00, batch count 58254, throughput 48.86 GB/s (read) Test completed: duration 10.00, batch count 60657, throughput 50.88 GB/s (write) ```

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 11, 2025):

added some timing code in get_buffer

std::shared_ptr<SliceBuffer> DistributedObjectStore::get_buffer(
    const std::string &key) {
    // Start high precision timing
    auto start_time = std::chrono::high_resolution_clock::now();
    
    if (!client_) {
        LOG(ERROR) << "Client is not initialized";
        return nullptr;
    }

    mooncake::Client::ObjectInfo object_info;
    SliceGuard guard(*this);  // Use SliceGuard for RAII
    uint64_t total_length = 0;
    ErrorCode error_code;
    std::shared_ptr<SliceBuffer> result = nullptr;

    // Timing point 1: Start Query operation
    auto query_start = std::chrono::high_resolution_clock::now();
    
    // Query the object info
    error_code = client_->Query(key, object_info);
    if (error_code == ErrorCode::OBJECT_NOT_FOUND) {
        return nullptr;
    }
    if (error_code != ErrorCode::OK) {
        LOG(ERROR) << "Query failed for key: " << key
                   << " with error: " << toString(error_code);
        return nullptr;
    }
    
    // Timing point 2: Query completed, start slice allocation
    auto query_end = std::chrono::high_resolution_clock::now();
    auto alloc_start = query_end;

    // Allocate slices for the object using the guard
    int ret = allocateSlices(guard.slices(), object_info, total_length);
    if (ret) {
        LOG(ERROR) << "Failed to allocate slices for key: " << key;
        return nullptr;
    }
    
    // Timing point 3: Allocation completed, start Get operation
    auto alloc_end = std::chrono::high_resolution_clock::now();
    auto get_start = alloc_end;

    // Get the object data
    error_code = client_->Get(key, object_info, guard.slices());
    if (error_code != ErrorCode::OK) {
        LOG(ERROR) << "Get failed for key: " << key
                   << " with error: " << toString(error_code);
        return nullptr;
    }
    
    // Timing point 4: Get completed, start buffer processing
    auto get_end = std::chrono::high_resolution_clock::now();
    auto process_start = get_end;

    if (guard.slices().size() == 1) {
        auto ptr = guard.slices()[0].ptr;
        guard.slices().clear();
        // Use SimpleAllocator for deallocation (default behavior)
        result = std::make_shared<SliceBuffer>(*this, ptr, total_length, true);
    } else {
        auto contiguous_buffer = exportSlices(guard.slices(), total_length);
        // Use delete[] for deallocation since exportSlices uses new char[]
        result = std::make_shared<SliceBuffer>(*this, contiguous_buffer,
                                               total_length, false);
    }
    
    // Timing point 5: Processing completed
    auto process_end = std::chrono::high_resolution_clock::now();
    auto total_end = process_end;
    
    // Calculate durations in microseconds
    auto query_duration = std::chrono::duration_cast<std::chrono::microseconds>(query_end - query_start);
    auto alloc_duration = std::chrono::duration_cast<std::chrono::microseconds>(alloc_end - alloc_start);
    auto get_duration = std::chrono::duration_cast<std::chrono::microseconds>(get_end - get_start);
    auto process_duration = std::chrono::duration_cast<std::chrono::microseconds>(process_end - process_start);
    auto total_duration = std::chrono::duration_cast<std::chrono::microseconds>(total_end - start_time);
    
    // Output timing results
    LOG(INFO) << "get_buffer timing for key '" << key << "' (size: " << total_length << " bytes):";
    LOG(INFO) << "  ├─ Query operation:     " << query_duration.count() << " μs";
    LOG(INFO) << "  ├─ Slice allocation:    " << alloc_duration.count() << " μs";
    LOG(INFO) << "  ├─ Get operation:       " << get_duration.count() << " μs";
    LOG(INFO) << "  ├─ Buffer processing:   " << process_duration.count() << " μs";
    LOG(INFO) << "  └─ Total time:          " << total_duration.count() << " μs";

    return result;
}

with chunk_size=128:

I20250611 19:49:45.947420 3454702 store_py.cpp:641]   ├─ Query operation:     156 μs
I20250611 19:49:45.947438 3454702 store_py.cpp:642]   ├─ Slice allocation:    2 μs
I20250611 19:49:45.947453 3454702 store_py.cpp:643]   ├─ Get operation:       673 μs
I20250611 19:49:45.947467 3454702 store_py.cpp:644]   ├─ Buffer processing:   4031 μs
I20250611 19:49:45.947484 3454702 store_py.cpp:645]   └─ Total time:          4865 μs
// get speed: 15.6MB/673 μs = 23.18GB/s
// overall speed = 3.20 GB/s

with chunk_size=16:

I20250611 19:56:36.884277 3456617 store_py.cpp:641]   ├─ Query operation:     116 μs
I20250611 19:56:36.884292 3456617 store_py.cpp:642]   ├─ Slice allocation:    1 μs
I20250611 19:56:36.884306 3456617 store_py.cpp:643]   ├─ Get operation:       298 μs
I20250611 19:56:36.884321 3456617 store_py.cpp:644]   ├─ Buffer processing:   0 μs
I20250611 19:56:36.884335 3456617 store_py.cpp:645]   └─ Total time:          417 μs
// get speed:  2M/298 μs = 6.71GB/s
// overall speed = 4.80 GB/s

Although chunk_size=16 avoids expensive memory operations, the actual performance is worse with this setting. I think this is partly due to the reduced transmission efficiency caused by smaller chunks, and partly because it significantly increases the overhead of integration with LMCache.

The fixed overhead of the query operation is also relatively high. Not sure if this can be merged with the get call into a single invocation.

The integration with LMCache also introduces some CPU-to-CPU, CPU-to-GPU transfers, as well as overhead from Python itself, which further lowers overall performance. For example, in this function apart from the C++ mooncake_get_buffer call, there are also time-consuming CPU memory allocation and copy operations afterward. These also show up in the timing results from the instrument connector, so the current output from the instrument connector seems reasonable.

@cicirori commented on GitHub (Jun 11, 2025): added some timing code in `get_buffer` ``` std::shared_ptr<SliceBuffer> DistributedObjectStore::get_buffer( const std::string &key) { // Start high precision timing auto start_time = std::chrono::high_resolution_clock::now(); if (!client_) { LOG(ERROR) << "Client is not initialized"; return nullptr; } mooncake::Client::ObjectInfo object_info; SliceGuard guard(*this); // Use SliceGuard for RAII uint64_t total_length = 0; ErrorCode error_code; std::shared_ptr<SliceBuffer> result = nullptr; // Timing point 1: Start Query operation auto query_start = std::chrono::high_resolution_clock::now(); // Query the object info error_code = client_->Query(key, object_info); if (error_code == ErrorCode::OBJECT_NOT_FOUND) { return nullptr; } if (error_code != ErrorCode::OK) { LOG(ERROR) << "Query failed for key: " << key << " with error: " << toString(error_code); return nullptr; } // Timing point 2: Query completed, start slice allocation auto query_end = std::chrono::high_resolution_clock::now(); auto alloc_start = query_end; // Allocate slices for the object using the guard int ret = allocateSlices(guard.slices(), object_info, total_length); if (ret) { LOG(ERROR) << "Failed to allocate slices for key: " << key; return nullptr; } // Timing point 3: Allocation completed, start Get operation auto alloc_end = std::chrono::high_resolution_clock::now(); auto get_start = alloc_end; // Get the object data error_code = client_->Get(key, object_info, guard.slices()); if (error_code != ErrorCode::OK) { LOG(ERROR) << "Get failed for key: " << key << " with error: " << toString(error_code); return nullptr; } // Timing point 4: Get completed, start buffer processing auto get_end = std::chrono::high_resolution_clock::now(); auto process_start = get_end; if (guard.slices().size() == 1) { auto ptr = guard.slices()[0].ptr; guard.slices().clear(); // Use SimpleAllocator for deallocation (default behavior) result = std::make_shared<SliceBuffer>(*this, ptr, total_length, true); } else { auto contiguous_buffer = exportSlices(guard.slices(), total_length); // Use delete[] for deallocation since exportSlices uses new char[] result = std::make_shared<SliceBuffer>(*this, contiguous_buffer, total_length, false); } // Timing point 5: Processing completed auto process_end = std::chrono::high_resolution_clock::now(); auto total_end = process_end; // Calculate durations in microseconds auto query_duration = std::chrono::duration_cast<std::chrono::microseconds>(query_end - query_start); auto alloc_duration = std::chrono::duration_cast<std::chrono::microseconds>(alloc_end - alloc_start); auto get_duration = std::chrono::duration_cast<std::chrono::microseconds>(get_end - get_start); auto process_duration = std::chrono::duration_cast<std::chrono::microseconds>(process_end - process_start); auto total_duration = std::chrono::duration_cast<std::chrono::microseconds>(total_end - start_time); // Output timing results LOG(INFO) << "get_buffer timing for key '" << key << "' (size: " << total_length << " bytes):"; LOG(INFO) << " ├─ Query operation: " << query_duration.count() << " μs"; LOG(INFO) << " ├─ Slice allocation: " << alloc_duration.count() << " μs"; LOG(INFO) << " ├─ Get operation: " << get_duration.count() << " μs"; LOG(INFO) << " ├─ Buffer processing: " << process_duration.count() << " μs"; LOG(INFO) << " └─ Total time: " << total_duration.count() << " μs"; return result; } ``` with `chunk_size=128`: ``` I20250611 19:49:45.947420 3454702 store_py.cpp:641] ├─ Query operation: 156 μs I20250611 19:49:45.947438 3454702 store_py.cpp:642] ├─ Slice allocation: 2 μs I20250611 19:49:45.947453 3454702 store_py.cpp:643] ├─ Get operation: 673 μs I20250611 19:49:45.947467 3454702 store_py.cpp:644] ├─ Buffer processing: 4031 μs I20250611 19:49:45.947484 3454702 store_py.cpp:645] └─ Total time: 4865 μs // get speed: 15.6MB/673 μs = 23.18GB/s // overall speed = 3.20 GB/s ``` with `chunk_size=16`: ``` I20250611 19:56:36.884277 3456617 store_py.cpp:641] ├─ Query operation: 116 μs I20250611 19:56:36.884292 3456617 store_py.cpp:642] ├─ Slice allocation: 1 μs I20250611 19:56:36.884306 3456617 store_py.cpp:643] ├─ Get operation: 298 μs I20250611 19:56:36.884321 3456617 store_py.cpp:644] ├─ Buffer processing: 0 μs I20250611 19:56:36.884335 3456617 store_py.cpp:645] └─ Total time: 417 μs // get speed: 2M/298 μs = 6.71GB/s // overall speed = 4.80 GB/s ``` Although `chunk_size=16` avoids expensive memory operations, the actual performance is worse with this setting. I think this is partly due to the reduced transmission efficiency caused by smaller chunks, and partly because it significantly increases the overhead of integration with LMCache. The fixed overhead of the query operation is also relatively high. Not sure if this can be merged with the get call into a single invocation. The integration with LMCache also introduces some CPU-to-CPU, CPU-to-GPU transfers, as well as overhead from Python itself, which further lowers overall performance. For example, in [this function](https://github.com/LMCache/LMCache/blob/9846a104b65f0dd7a93419a0d2da99b4017a7380/lmcache/v1/storage_backend/connector/mooncakestore_connector.py#L129) apart from the C++ `mooncake_get_buffer` call, there are also time-consuming CPU memory allocation and copy operations afterward. These also show up in the timing results from the instrument connector, so the current output from the instrument connector seems reasonable.

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 12, 2025):

I'm working on the batch interface here: https://github.com/kvcache-ai/Mooncake/pull/455. Once our implementation is ready, I’ll submit a PR to lmcache to integrate this batch API.

In my tests, the batch interface can saturate the NIC bandwidth, so performance looks promising.

When the value size exceeds 16,777,200 bytes (kMaxSliceSize), it gets split up. In the end, since tensors need to be in contiguous memory, we have to stitch slices back together. But for a 15.6MB value, that copying shouldn't be happening—I'll dig into it a bit more. Maybe try lowering the value size to 14MB and see if it helps?

Also, exportSlices just allocates memory and does memcpy, but it's taking around 4ms, which seems a bit high—I should probably look into that too.

I'll share my test model and config with you shortly, so you can reproduce the performance more easily on your side.

@xiaguan commented on GitHub (Jun 12, 2025): I'm working on the batch interface here: https://github.com/kvcache-ai/Mooncake/pull/455. Once our implementation is ready, I’ll submit a PR to lmcache to integrate this batch API. In my tests, the batch interface can saturate the NIC bandwidth, so performance looks promising. When the value size exceeds 16,777,200 bytes (kMaxSliceSize), it gets split up. In the end, since tensors need to be in contiguous memory, we have to stitch slices back together. But for a 15.6MB value, that copying shouldn't be happening—I'll dig into it a bit more. Maybe try lowering the value size to 14MB and see if it helps? Also, exportSlices just allocates memory and does memcpy, but it's taking around 4ms, which seems a bit high—I should probably look into that too. I'll share my test model and config with you shortly, so you can reproduce the performance more easily on your side.

bsakaguchi commented

2026-10-29 21:10:22 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 12, 2025):

lmcache config

chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://127.0.0.1:50051/"
remote_serde: "naive"
pipelined_backend: False
local_cpu: False
max_local_cpu_size: 5

mooncake( i used main branch code)

{
    "local_hostname": "localhost",
    "metadata_server": "http://localhost:8080/metadata",
    "protocol": "rdma",
    "device_name": "nic",
    "global_segment_size": 10737418240,
    "master_server_address": "localhost:50051",
    "local_buffer_size": 5368709120
}

vllm

#!/bin/bash

# Set all your environment variables
export CUDA_VISIBLE_DEVICES=4,5,6,7
export MOONCAKE_CONFIG_PATH=/mooncake.json
export LMCACHE_USE_EXPERIMENTAL=True
export MC_LOG_LEVEL=TRACE
export MC_TE_METRIC=1
export LMCACHE_TRACK_USAGE=false
export VLLM_MLA_DISABLE=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export VLLM_USE_V1=1
export LMCACHE_CONFIG_FILE=/lmcache-mooncake.yaml

# Then execute your command with numactl
vllm serve DeepSeek-R1-Distill-Qwen-7B \
            --trust-remote-code \
            --served-model-name vllm_cpu_offload \
            --max-model-len 24784 \
            --max-seq-len-to-capture 10000 \
            --max-num-seqs 64 \
            --gpu-memory-utilization 0.9 \
            --no-enable-chunked-prefill \
            --no-enable-prefix-caching \
            --disable-log-requests \
            --host 0.0.0.0 \
            -tp 4 \
            --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' --enforce-eager

how i bench

#!/bin/bash

# Define the common command
CMD="python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --model \"DeepSeek-R1-Distill-Qwen-7B\" \
    --served-model-name \"vllm_cpu_offload\" \
    --dataset-name random \
    --random-input-len 10000 \
    --random-output-len 128 \
    --num-prompts 30 \
    --seed 42 \
    --request-rate inf \
    --trust-remote-code \
    --host 127.0.0.1 \
    --port 8000 \
    --endpoint \"/v1/completions\""

# First run (populating cache, if any)
echo "--- Starting First Run (Cache Miss Expected) ---"
eval $CMD
echo "--- First Run Completed ---"

echo # Add a blank line for readability

# Second run (testing cache)
echo "--- Starting Second Run (Cache Hit Expected) ---"
eval $CMD
echo "--- Second Run Completed ---"

The results i get

============ Serving Benchmark Result ============
Successful requests:                     30        
Benchmark duration (s):                  35.27     
Total input tokens:                      299970    
Total generated tokens:                  3840      
Request throughput (req/s):              0.85      
Output token throughput (tok/s):         108.86    
Total Token throughput (tok/s):          8613.08   
---------------Time to First Token----------------
Mean TTFT (ms):                          16442.13  
Median TTFT (ms):                        16194.19  
P99 TTFT (ms):                           32801.94  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.05    
Median TPOT (ms):                        141.76    
P99 TPOT (ms):                           231.62    
---------------Inter-token Latency----------------
Mean ITL (ms):                           138.96    
Median ITL (ms):                         214.71    
P99 ITL (ms):                            265.04    
==================================================
--- First Run Completed ---

--- Starting Second Run (Cache Hit Expected) ---
============ Serving Benchmark Result ============
Successful requests:                     30        
Benchmark duration (s):                  8.10      
Total input tokens:                      299970    
Total generated tokens:                  3840      
Request throughput (req/s):              3.71      
Output token throughput (tok/s):         474.33    
Total Token throughput (tok/s):          37527.65  
---------------Time to First Token----------------
Mean TTFT (ms):                          3611.85   
Median TTFT (ms):                        5191.99   
P99 TTFT (ms):                           5200.80   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          35.13     
Median TPOT (ms):                        22.77     
P99 TPOT (ms):                           61.46     
---------------Inter-token Latency----------------
Mean ITL (ms):                           35.13     
Median ITL (ms):                         22.48     
P99 ITL (ms):                            30.74     
==================================================
--- Second Run Completed ---

@xiaguan commented on GitHub (Jun 12, 2025): lmcache config ``` chunk_size: 256 local_device: "cpu" remote_url: "mooncakestore://127.0.0.1:50051/" remote_serde: "naive" pipelined_backend: False local_cpu: False max_local_cpu_size: 5 ``` mooncake( i used main branch code) ``` { "local_hostname": "localhost", "metadata_server": "http://localhost:8080/metadata", "protocol": "rdma", "device_name": "nic", "global_segment_size": 10737418240, "master_server_address": "localhost:50051", "local_buffer_size": 5368709120 } ``` vllm ``` #!/bin/bash # Set all your environment variables export CUDA_VISIBLE_DEVICES=4,5,6,7 export MOONCAKE_CONFIG_PATH=/mooncake.json export LMCACHE_USE_EXPERIMENTAL=True export MC_LOG_LEVEL=TRACE export MC_TE_METRIC=1 export LMCACHE_TRACK_USAGE=false export VLLM_MLA_DISABLE=1 export VLLM_ATTENTION_BACKEND=FLASH_ATTN export VLLM_USE_V1=1 export LMCACHE_CONFIG_FILE=/lmcache-mooncake.yaml # Then execute your command with numactl vllm serve DeepSeek-R1-Distill-Qwen-7B \ --trust-remote-code \ --served-model-name vllm_cpu_offload \ --max-model-len 24784 \ --max-seq-len-to-capture 10000 \ --max-num-seqs 64 \ --gpu-memory-utilization 0.9 \ --no-enable-chunked-prefill \ --no-enable-prefix-caching \ --disable-log-requests \ --host 0.0.0.0 \ -tp 4 \ --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' --enforce-eager ``` how i bench ``` #!/bin/bash # Define the common command CMD="python3 vllm/benchmarks/benchmark_serving.py \ --backend vllm \ --model \"DeepSeek-R1-Distill-Qwen-7B\" \ --served-model-name \"vllm_cpu_offload\" \ --dataset-name random \ --random-input-len 10000 \ --random-output-len 128 \ --num-prompts 30 \ --seed 42 \ --request-rate inf \ --trust-remote-code \ --host 127.0.0.1 \ --port 8000 \ --endpoint \"/v1/completions\"" # First run (populating cache, if any) echo "--- Starting First Run (Cache Miss Expected) ---" eval $CMD echo "--- First Run Completed ---" echo # Add a blank line for readability # Second run (testing cache) echo "--- Starting Second Run (Cache Hit Expected) ---" eval $CMD echo "--- Second Run Completed ---" ``` The results i get ``` ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 35.27 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 0.85 Output token throughput (tok/s): 108.86 Total Token throughput (tok/s): 8613.08 ---------------Time to First Token---------------- Mean TTFT (ms): 16442.13 Median TTFT (ms): 16194.19 P99 TTFT (ms): 32801.94 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 138.05 Median TPOT (ms): 141.76 P99 TPOT (ms): 231.62 ---------------Inter-token Latency---------------- Mean ITL (ms): 138.96 Median ITL (ms): 214.71 P99 ITL (ms): 265.04 ================================================== --- First Run Completed --- --- Starting Second Run (Cache Hit Expected) --- ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 8.10 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 3.71 Output token throughput (tok/s): 474.33 Total Token throughput (tok/s): 37527.65 ---------------Time to First Token---------------- Mean TTFT (ms): 3611.85 Median TTFT (ms): 5191.99 P99 TTFT (ms): 5200.80 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.13 Median TPOT (ms): 22.77 P99 TPOT (ms): 61.46 ---------------Inter-token Latency---------------- Mean ITL (ms): 35.13 Median ITL (ms): 22.48 P99 ITL (ms): 30.74 ================================================== --- Second Run Completed --- ```

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 12, 2025):

I'm working on the batch interface here: https://github.com/kvcache-ai/Mooncake/pull/455. Once our implementation is ready, I’ll submit a PR to lmcache to integrate this batch API.

that's nice! thanks for your great work and make it open sourced.

When the value size exceeds 16,777,200 bytes (kMaxSliceSize), it gets split up.

github.com/facebook/CacheLib@9005ef4224/cachelib/allocator/memory/Slab.h (L86)
Have you made any changes to cachelib? According to the open source version it should be 4MB?
BTW I'm reproducing your results and would like to know on what GPU model you did the test~

@cicirori commented on GitHub (Jun 12, 2025): > I'm working on the batch interface here: https://github.com/kvcache-ai/Mooncake/pull/455. Once our implementation is ready, I’ll submit a PR to lmcache to integrate this batch API. that's nice! thanks for your great work and make it open sourced. > When the value size exceeds 16,777,200 bytes (kMaxSliceSize), it gets split up. https://github.com/facebook/CacheLib/blob/9005ef4224f84678f47a476c2225c930e391bfe4/cachelib/allocator/memory/Slab.h#L86 Have you made any changes to cachelib? According to the open source version it should be 4MB? BTW I'm reproducing your results and would like to know on what GPU model you did the test~

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 12, 2025):

tried to reproduce with H100
without lmcache:

============ Serving Benchmark Result ============
Successful requests:                     30
Benchmark duration (s):                  3.90
Total input tokens:                      299970
Total generated tokens:                  3840
Request throughput (req/s):              7.69
Output token throughput (tok/s):         983.75
Total Token throughput (tok/s):          77831.60
---------------Time to First Token----------------
Mean TTFT (ms):                          1538.59
Median TTFT (ms):                        1532.97
P99 TTFT (ms):                           2794.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.06
Median TPOT (ms):                        17.11
P99 TPOT (ms):                           25.50
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.06
Median ITL (ms):                         7.22
P99 ITL (ms):                            77.62
==================================================
--- First Run Completed ---

--- Starting Second Run (Cache Hit Expected) ---
============ Serving Benchmark Result ============
Successful requests:                     30
Benchmark duration (s):                  3.80
Total input tokens:                      299970
Total generated tokens:                  3840
Request throughput (req/s):              7.90
Output token throughput (tok/s):         1010.87
Total Token throughput (tok/s):          79977.27
---------------Time to First Token----------------
Mean TTFT (ms):                          1539.96
Median TTFT (ms):                        1527.34
P99 TTFT (ms):                           2785.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.54
Median TPOT (ms):                        16.53
P99 TPOT (ms):                           25.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           16.54
Median ITL (ms):                         7.43
P99 ITL (ms):                            76.85
==================================================
--- Second Run Completed ---

with lmcache x mooncake:

============ Serving Benchmark Result ============
Successful requests:                     30
Benchmark duration (s):                  6.00
Total input tokens:                      299970
Total generated tokens:                  3840
Request throughput (req/s):              5.00
Output token throughput (tok/s):         640.52
Total Token throughput (tok/s):          50676.45
---------------Time to First Token----------------
Mean TTFT (ms):                          2593.78
Median TTFT (ms):                        2599.13
P99 TTFT (ms):                           4773.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.08
Median TPOT (ms):                        25.00
P99 TPOT (ms):                           40.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         7.43
P99 ITL (ms):                            150.14
==================================================
--- First Run Completed ---

--- Starting Second Run (Cache Hit Expected) ---
============ Serving Benchmark Result ============
Successful requests:                     30
Benchmark duration (s):                  6.05
Total input tokens:                      299970
Total generated tokens:                  3840
Request throughput (req/s):              4.96
Output token throughput (tok/s):         634.72
Total Token throughput (tok/s):          50217.12
---------------Time to First Token----------------
Mean TTFT (ms):                          4018.46
Median TTFT (ms):                        4881.53
P99 TTFT (ms):                           4909.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.73
Median TPOT (ms):                        8.90
P99 TPOT (ms):                           42.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           15.73
Median ITL (ms):                         7.42
P99 ITL (ms):                            24.58
==================================================
--- Second Run Completed ---

@cicirori commented on GitHub (Jun 12, 2025): tried to reproduce with H100 without lmcache: ``` ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 3.90 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 7.69 Output token throughput (tok/s): 983.75 Total Token throughput (tok/s): 77831.60 ---------------Time to First Token---------------- Mean TTFT (ms): 1538.59 Median TTFT (ms): 1532.97 P99 TTFT (ms): 2794.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.06 Median TPOT (ms): 17.11 P99 TPOT (ms): 25.50 ---------------Inter-token Latency---------------- Mean ITL (ms): 17.06 Median ITL (ms): 7.22 P99 ITL (ms): 77.62 ================================================== --- First Run Completed --- --- Starting Second Run (Cache Hit Expected) --- ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 3.80 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 7.90 Output token throughput (tok/s): 1010.87 Total Token throughput (tok/s): 79977.27 ---------------Time to First Token---------------- Mean TTFT (ms): 1539.96 Median TTFT (ms): 1527.34 P99 TTFT (ms): 2785.03 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.54 Median TPOT (ms): 16.53 P99 TPOT (ms): 25.48 ---------------Inter-token Latency---------------- Mean ITL (ms): 16.54 Median ITL (ms): 7.43 P99 ITL (ms): 76.85 ================================================== --- Second Run Completed --- ``` with lmcache x mooncake: ``` ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 6.00 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 5.00 Output token throughput (tok/s): 640.52 Total Token throughput (tok/s): 50676.45 ---------------Time to First Token---------------- Mean TTFT (ms): 2593.78 Median TTFT (ms): 2599.13 P99 TTFT (ms): 4773.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.08 Median TPOT (ms): 25.00 P99 TPOT (ms): 40.03 ---------------Inter-token Latency---------------- Mean ITL (ms): 25.08 Median ITL (ms): 7.43 P99 ITL (ms): 150.14 ================================================== --- First Run Completed --- --- Starting Second Run (Cache Hit Expected) --- ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 6.05 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 4.96 Output token throughput (tok/s): 634.72 Total Token throughput (tok/s): 50217.12 ---------------Time to First Token---------------- Mean TTFT (ms): 4018.46 Median TTFT (ms): 4881.53 P99 TTFT (ms): 4909.35 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.73 Median TPOT (ms): 8.90 P99 TPOT (ms): 42.61 ---------------Inter-token Latency---------------- Mean ITL (ms): 15.73 Median ITL (ms): 7.42 P99 ITL (ms): 24.58 ================================================== --- Second Run Completed --- ```

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 12, 2025):

Just to give you a quick update on the current status and some performance notes:

My gpu model is A10 GPU.
We've modified cachelib since the original 4MB cache size was a bit too small for KV cache scenarios.
Looks like your GPU is quite powerful for this model, which might explain some of the performance differences.

To summarize:

Mooncake is a distributed storage layer built on top of the Transfer Engine. With large enough batch sizes, it can push throughput close to the NIC bandwidth limit(as the transfer engine bench result). So we can’t really go faster than what the NIC allows.
Regarding memcpy: once this PR is merged (https://github.com/LMCache/LMCache/pull/821), I’ll submit a follow-up to reduce copies in get().
For query latency, I’m working on a client-side master meta cache. This should help. If batch still suffers from query latency.
On the batch_get/batch_put side in LMCache: the C++ implementation is already done, since that PR is merged(https://github.com/kvcache-ai/Mooncake/pull/455). I’ll expose the Python interface soon and start using it in LMCache.

That’s a quick snapshot of where things are at and the current perf-related work.
You could also join our slack https://join.slack.com/t/mooncake-project/shared_invite/zt-365h0e2tz-rfXTibSuGKIzik3GUYBJbg

@xiaguan commented on GitHub (Jun 12, 2025): Just to give you a quick update on the current status and some performance notes: * My gpu model is A10 GPU. * We've modified cachelib since the original 4MB cache size was a bit too small for KV cache scenarios. * Looks like your GPU is quite powerful for this model, which might explain some of the performance differences. To summarize: * Mooncake is a distributed storage layer built on top of the Transfer Engine. With large enough batch sizes, it can push throughput close to the NIC bandwidth limit(as the transfer engine bench result). So we can’t really go faster than what the NIC allows. * Regarding memcpy: once this PR is merged (https://github.com/LMCache/LMCache/pull/821), I’ll submit a follow-up to reduce copies in get(). * For query latency, I’m working on a client-side master meta cache. This should help. If batch still suffers from query latency. * On the batch_get/batch_put side in LMCache: the C++ implementation is already done, since that PR is merged(https://github.com/kvcache-ai/Mooncake/pull/455). I’ll expose the Python interface soon and start using it in LMCache. That’s a quick snapshot of where things are at and the current perf-related work. You could also join our slack https://join.slack.com/t/mooncake-project/shared_invite/zt-365h0e2tz-rfXTibSuGKIzik3GUYBJbg

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 13, 2025):

I think I found the reason why Mooncake is running slower.

For GDS and Local, their exist() or contains() interface just does a quick in-memory lookup (like checking a hashtable or dict), so it's super fast. But Mooncake calls an RPC to the master node instead — even though that RPC is pretty fast (around 100µs), it looks like the scheduler is sensitive to that extra latency.

I did a quick hack where I made Mooncake use a local set for exist(), and that seemed to help.

============ Serving Benchmark Result ============
Successful requests:                     30        
Benchmark duration (s):                  34.65     
Total input tokens:                      299970    
Total generated tokens:                  3840      
Request throughput (req/s):              0.87      
Output token throughput (tok/s):         110.84    
Total Token throughput (tok/s):          8769.03   
---------------Time to First Token----------------
Mean TTFT (ms):                          16024.46  
Median TTFT (ms):                        15722.36  
P99 TTFT (ms):                           32206.26  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          136.26    
Median TPOT (ms):                        140.74    
P99 TPOT (ms):                           227.76    
---------------Inter-token Latency----------------
Mean ITL (ms):                           137.05    
Median ITL (ms):                         212.32    
P99 ITL (ms):                            252.49    
==================================================
--- First Run Completed ---
============ Serving Benchmark Result ============
Successful requests:                     30        
Benchmark duration (s):                  4.94      
Total input tokens:                      299970    
Total generated tokens:                  3840      
Request throughput (req/s):              6.08      
Output token throughput (tok/s):         777.86    
Total Token throughput (tok/s):          61541.88  
---------------Time to First Token----------------
Mean TTFT (ms):                          1445.25   
Median TTFT (ms):                        1862.26   
P99 TTFT (ms):                           2064.08   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          27.23     
Median TPOT (ms):                        24.02     
P99 TPOT (ms):                           36.70     
---------------Inter-token Latency----------------
Mean ITL (ms):                           27.23     
Median ITL (ms):                         22.48     
P99 ITL (ms):                            202.97

Much better then previos bench.

We're planning to use the master's lease mechanism to build a client-side metadata cache to help reduce the impact of this issue.

Another idea we're considering is implementing something like batch_exists in the lmcache layer to better hide the network latency.

@xiaguan commented on GitHub (Jun 13, 2025): I think I found the reason why Mooncake is running slower. For GDS and Local, their `exist()` or `contains()` interface just does a quick in-memory lookup (like checking a hashtable or dict), so it's super fast. But Mooncake calls an RPC to the master node instead — even though that RPC is pretty fast (around 100µs), it looks like the scheduler is sensitive to that extra latency. I did a quick hack where I made Mooncake use a local set for exist(), and that seemed to help. ``` ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 34.65 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 0.87 Output token throughput (tok/s): 110.84 Total Token throughput (tok/s): 8769.03 ---------------Time to First Token---------------- Mean TTFT (ms): 16024.46 Median TTFT (ms): 15722.36 P99 TTFT (ms): 32206.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 136.26 Median TPOT (ms): 140.74 P99 TPOT (ms): 227.76 ---------------Inter-token Latency---------------- Mean ITL (ms): 137.05 Median ITL (ms): 212.32 P99 ITL (ms): 252.49 ================================================== --- First Run Completed --- ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 4.94 Total input tokens: 299970 Total generated tokens: 3840 Request throughput (req/s): 6.08 Output token throughput (tok/s): 777.86 Total Token throughput (tok/s): 61541.88 ---------------Time to First Token---------------- Mean TTFT (ms): 1445.25 Median TTFT (ms): 1862.26 P99 TTFT (ms): 2064.08 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 27.23 Median TPOT (ms): 24.02 P99 TPOT (ms): 36.70 ---------------Inter-token Latency---------------- Mean ITL (ms): 27.23 Median ITL (ms): 22.48 P99 ITL (ms): 202.97 ``` Much better then previos bench. We're planning to use the master's lease mechanism to build a client-side metadata cache to help reduce the impact of this issue. Another idea we're considering is implementing something like batch_exists in the lmcache layer to better hide the network latency.

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 13, 2025):

This is one of the reasons. I happened to benchmark this interface as well and confirmed the performance is indeed around 100μs, but I haven’t done deeper investigation yet.

I've made some optimizations on my side and saw significant improvements in my environment:

Removed the fast path for client get, which uses memcpy. RDMA is actually faster here. This logic doesn’t seem to exist on the main branch, but the commit I'm using still has it.
Based on the LMCache PR you shared earlier, I registered LMCache's CPU buffer to the Mooncake store, enabling direct transfer to that region and achieving zero-copy on the CPU side.
Tuned some parameters to ensure the baseline transfer speed is solid.

I haven’t looked into optimizations for query and exist yet, but with the above changes, the performance in my environment improved ~7x. In some H100 cases, it's already faster than recomputation. I believe with further optimizations for query and exist, along with zero-copy on the GPU side, the performance of Mooncake Store X LMCache will be quite impressive. Thanks for open-sourcing this project!

updated result from my original test:

	first run(kv cache store)	second run(kv cache retrieve)
baseline	8.46s	8.47s
mooncake(chunk size 256)	12.10s	29.00s
gds	14.04s	2.70s
local dram	9.42s	1.77s
mooncake(chunk size 16)	34.13s	48.72s
mooncake(chunk size 128 + etcd)	11.94s	27.67s
mooncake after perf	9.94s	3.71s

~20s ->~10s Eliminated redundant memory copy inside Mooncake Store and tuned some parameters
~10s -> 4s Implement CPU zero copy between LMCache and Mooncake store.
4s -> 3.71s Apply the latest sync contains pr

Of course these numbers are very workload and environment dependent, just posting here to show some of the bottlenecks I see.

@cicirori commented on GitHub (Jun 13, 2025): This is one of the reasons. I happened to benchmark this interface as well and confirmed the performance is indeed around 100μs, but I haven’t done deeper investigation yet. I've made some optimizations on my side and saw significant improvements in my environment: 1. Removed the fast path for client get, which uses `memcpy`. RDMA is actually faster here. This logic doesn’t seem to exist on the main branch, but the commit I'm using still has it. 2. Based on the LMCache PR you shared earlier, I registered LMCache's CPU buffer to the Mooncake store, enabling direct transfer to that region and achieving zero-copy on the CPU side. 3. Tuned some parameters to ensure the baseline transfer speed is solid. I haven’t looked into optimizations for `query` and `exist` yet, but with the above changes, the performance in my environment improved ~7x. In some H100 cases, it's already faster than recomputation. I believe with further optimizations for `query` and `exist`, along with zero-copy on the GPU side, the performance of Mooncake Store X LMCache will be quite impressive. Thanks for open-sourcing this project! updated result from my original test: | first run(kv cache store) | second run(kv cache retrieve) -- | -- | -- baseline | 8.46s | 8.47s mooncake(chunk size 256) | 12.10s | 29.00s gds | 14.04s | 2.70s local dram | 9.42s | 1.77s mooncake(chunk size 16) | 34.13s | 48.72s mooncake(chunk size 128 + etcd) | 11.94s | 27.67s mooncake after perf | 9.94s | 3.71s 1. ~20s ->~10s Eliminated redundant memory copy inside Mooncake Store and tuned some parameters 2. ~10s -> 4s Implement CPU zero copy between LMCache and Mooncake store. 3. 4s -> 3.71s Apply the latest sync contains pr Of course these numbers are very workload and environment dependent, just posting here to show some of the bottlenecks I see.

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 13, 2025):

Outstanding work! It's exciting to see Mooncake, as a distributed storage solution, delivering such excellent performance.

With my 100Gbps RDMA NIC, I’ve noticed that memcpy is actually faster than RDMA. Perhaps I should make this behavior configurable. I'm just curious—why is RDMA typically faster than memcpy? I'm wondering about the underlying reason.

Welcome to contribute to both Mooncake!

@xiaguan commented on GitHub (Jun 13, 2025): Outstanding work! It's exciting to see Mooncake, as a distributed storage solution, delivering such excellent performance. With my 100Gbps RDMA NIC, I’ve noticed that memcpy is actually faster than RDMA. Perhaps I should make this behavior configurable. I'm just curious—why is RDMA typically faster than memcpy? I'm wondering about the underlying reason. Welcome to contribute to both Mooncake!

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@cicirori commented on GitHub (Jun 13, 2025):

My NIC is 400Gbs, and this may be related to the experimental environment, my simple understanding is that memcpy will be subject to the current system load and some other interference, go RDMA only need to give two pointers, the transfer process bypasses the linux kernel stack also do not need to involve the CPU, the speed will be more stable.

Here is a very simple and naive memcpy_test result, it should be worse if run under real load:

--- Benchmarking memcpy Speed ---
---------------------------------
Size:     0.00 MB | Time: 0.000000 s | Speed: 7.91 GB/s
Size:     0.01 MB | Time: 0.000000 s | Speed: 27.54 GB/s
Size:     0.02 MB | Time: 0.000000 s | Speed: 31.20 GB/s
Size:     0.03 MB | Time: 0.000001 s | Speed: 20.93 GB/s
Size:     0.06 MB | Time: 0.000004 s | Speed: 17.33 GB/s
Size:     0.12 MB | Time: 0.000006 s | Speed: 19.17 GB/s
Size:     0.25 MB | Time: 0.000011 s | Speed: 21.86 GB/s
Size:     0.50 MB | Time: 0.000022 s | Speed: 22.32 GB/s
Size:     1.00 MB | Time: 0.000070 s | Speed: 14.03 GB/s
Size:     2.00 MB | Time: 0.000225 s | Speed: 8.69 GB/s
Size:     4.00 MB | Time: 0.000489 s | Speed: 7.98 GB/s
Size:     8.00 MB | Time: 0.000867 s | Speed: 9.01 GB/s
Size:    16.00 MB | Time: 0.001576 s | Speed: 9.91 GB/s
Size:    32.00 MB | Time: 0.001701 s | Speed: 18.37 GB/s
Size:    64.00 MB | Time: 0.004036 s | Speed: 15.48 GB/s
Size:   128.00 MB | Time: 0.008736 s | Speed: 14.31 GB/s
Size:   256.00 MB | Time: 0.018426 s | Speed: 13.57 GB/s

bsakaguchi commented

2026-10-29 21:10:23 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 8, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 8, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:24 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:46 +00:00

[PR #163] [MERGED] [DOC] remove MULAN license to ease vllm deployment #1364

Rows
Columns

[GH-ISSUE #467] mooncake X LMCache performance #163