[GH-ISSUE #3658] [Bug]: batch_get_session_start fails with 702 INVALID_REPLICA when SSD offloading is enabled #934

New issue

Closed

opened 2026-10-29 21:21:04 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:21:04 +00:00

Owner

Originally created by @Yong-ee on GitHub (Aug 25, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3658

Bug Report

Description

I deployed Mooncake master and client using the latest main branch, with SSD offloading enabled. When retrieving data via the Get Session API, I received error 702 (ErrorCode::INVALID_REPLICA). The logs show the following errors:

E0825 11:48:28.436453 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_9ef0d3fc87769163f4a9ca7132f99420d47e3b40eb000648fcae5b788a6f0f65_tp3_cp0_pp0_deepseek_v4_c4
E0825 11:48:28.436558 675219 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_334485bcb2be8214beda12734ad46ac3ac9a8c71c7c33b81a184ce65e58df9f2_tp6_cp0_pp0_deepseek_v4_c4
E0825 11:48:28.436663 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_49d5dfdaf24abeb87994a10f24550206352df5608833921098d6e74e6dcda9f3_tp3_cp0_pp0_deepseek_v4_c4
E0825 11:48:28.436770 675219 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_501666f1aa9c91c993cd32cdb569d2d6d1a01217c2c044a301cdbda2cebb40b9_tp6_cp0_pp0_deepseek_v4_c4
E0825 11:48:28.436877 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_c80468b0bfe1208838299de45678e9e9c16cfc13a7715bc7394a109a60d794bf_tp3_cp0_pp0_deepseek_v4_c4

Root Cause

When SSD offloading is enabled, some data resides on SSD instead of in‑memory.
The batch_get_session_start method only selects memory replicas. As a result, keys whose replicas exist only on SSD will return error 702.

std::vector<int> RealClient::batch_get_session_start(
    const std::vector<std::string> &keys) {
std::vector<int> results(
keys.size(), static_cast<int>(toInt(ErrorCode::INVALID_PARAMS)));
if (!client_) {
LOG(ERROR) << "Client is not initialized";
return results;
    }
if (keys.empty()) {
return {};
    }
    // Master interaction only here: query replicas + lease.
const auto query_results = client_->BatchQuery(keys);
auto local_endpoints = client_->GetLocalEndpoints();
std::lock_guard<std::mutex> lock(session_mutex_);
for (size_t i = 0; i < keys.size(); ++i) {
if (!query_results[i]) {
results[i] = static_cast<int>(toInt(query_results[i].error()));
get_sessions_.erase(keys[i]);
continue;
        }
auto query_result = query_results[i].value();
if (query_result.IsLeaseExpired()) {
results[i] = static_cast<int>(toInt(ErrorCode::LEASE_EXPIRED));
get_sessions_.erase(keys[i]);
continue;
        }
const auto *replica =
SelectCompleteMemoryReplica(query_result.replicas, local_endpoints);
if (!replica) {
LOG(ERROR) << "No complete memory replica for key: " << keys[i];
results[i] = static_cast<int>(toInt(ErrorCode::INVALID_REPLICA));
get_sessions_.erase(keys[i]);
continue;
        }
        // QueryResult members are const: erase + emplace (no operator=).
get_sessions_.erase(keys[i]);
get_sessions_.emplace(keys[i],
FilterQueryResult(query_result, *replica));
results[i] = 0;
    }
return results;
}

Expected behavior

Similar to batch_get_into, use SelectBestReplica to select the best replica for each key.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Yong-ee on GitHub (Aug 25, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3658 ### Bug Report ### Description I deployed Mooncake master and client using the latest main branch, with SSD offloading enabled. When retrieving data via the Get Session API, I received error `702 (ErrorCode::INVALID_REPLICA)`. The logs show the following errors: ``` E0825 11:48:28.436453 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_9ef0d3fc87769163f4a9ca7132f99420d47e3b40eb000648fcae5b788a6f0f65_tp3_cp0_pp0_deepseek_v4_c4 E0825 11:48:28.436558 675219 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_334485bcb2be8214beda12734ad46ac3ac9a8c71c7c33b81a184ce65e58df9f2_tp6_cp0_pp0_deepseek_v4_c4 E0825 11:48:28.436663 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_49d5dfdaf24abeb87994a10f24550206352df5608833921098d6e74e6dcda9f3_tp3_cp0_pp0_deepseek_v4_c4 E0825 11:48:28.436770 675219 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_501666f1aa9c91c993cd32cdb569d2d6d1a01217c2c044a301cdbda2cebb40b9_tp6_cp0_pp0_deepseek_v4_c4 E0825 11:48:28.436877 675216 real_client.cpp:5005] No complete memory replica for key: -models-DeepSeek-V4-Flash-0731-Channel-INT8-w8a8_c80468b0bfe1208838299de45678e9e9c16cfc13a7715bc7394a109a60d794bf_tp3_cp0_pp0_deepseek_v4_c4 ``` ### Root Cause When SSD offloading is enabled, some data resides on SSD instead of in‑memory. The `batch_get_session_start` method only selects memory replicas. As a result, keys whose replicas exist only on SSD will return error 702. ``` std::vector<int> RealClient::batch_get_session_start( const std::vector<std::string> &keys) { std::vector<int> results( keys.size(), static_cast<int>(toInt(ErrorCode::INVALID_PARAMS))); if (!client_) { LOG(ERROR) << "Client is not initialized"; return results; } if (keys.empty()) { return {}; } // Master interaction only here: query replicas + lease. const auto query_results = client_->BatchQuery(keys); auto local_endpoints = client_->GetLocalEndpoints(); std::lock_guard<std::mutex> lock(session_mutex_); for (size_t i = 0; i < keys.size(); ++i) { if (!query_results[i]) { results[i] = static_cast<int>(toInt(query_results[i].error())); get_sessions_.erase(keys[i]); continue; } auto query_result = query_results[i].value(); if (query_result.IsLeaseExpired()) { results[i] = static_cast<int>(toInt(ErrorCode::LEASE_EXPIRED)); get_sessions_.erase(keys[i]); continue; } const auto *replica = SelectCompleteMemoryReplica(query_result.replicas, local_endpoints); if (!replica) { LOG(ERROR) << "No complete memory replica for key: " << keys[i]; results[i] = static_cast<int>(toInt(ErrorCode::INVALID_REPLICA)); get_sessions_.erase(keys[i]); continue; } // QueryResult members are const: erase + emplace (no operator=). get_sessions_.erase(keys[i]); get_sessions_.emplace(keys[i], FilterQueryResult(query_result, *replica)); results[i] = 0; } return results; } ``` ### Expected behavior Similar to `batch_get_into`, use `SelectBestReplica` to select the best replica for each key. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:21:04 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:21:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 25, 2026):

Thanks for opening this issue, @Yong-ee!

Field	Value
Issue	#3658
GitHub user ID	`15357666`
Reporter	@Yong-ee

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 25, 2026): Thanks for opening this issue, @Yong-ee! | Field | Value | |-------|-------| | **Issue** | #3658 | | **GitHub user ID** | `15357666` | | **Reporter** | @Yong-ee | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:21:05 +00:00

Author

Owner

@Yong-ee commented on GitHub (Aug 25, 2026):

cc @ascend-direct-dev

@Yong-ee commented on GitHub (Aug 25, 2026): cc @ascend-direct-dev

bsakaguchi commented

2026-10-29 21:21:06 +00:00

Author

Owner

@Yong-ee commented on GitHub (Aug 25, 2026):

https://github.com/kvcache-ai/Mooncake/pull/3662 this PR, batch_get_into_multi_buffer_ranges does not adapt to non‑memory replicas. Therefore, if a non‑memory replica is selected, get‑buffer operations will still encounter issues.

@Yong-ee commented on GitHub (Aug 25, 2026): https://github.com/kvcache-ai/Mooncake/pull/3662 this PR, `batch_get_into_multi_buffer_ranges` does not adapt to non‑memory replicas. Therefore, if a non‑memory replica is selected, get‑buffer operations will still encounter issues.

bsakaguchi referenced this issue

2026-10-29 21:31:19 +00:00

[PR #934] [CLOSED] [TransferEngine] Conditionally enable SO_REUSEADDR in handshake daemon #1912