kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2636] [Bug]: Snapshot restore loses segment protocol and allocator replica_type #691

New issue

Closed

opened 2026-10-29 21:18:15 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:18:15 +00:00

Owner

Originally created by @mzygQAQ on GitHub (Jun 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2636

Originally assigned to: @mzygQAQ, @Icedcoco on GitHub.

Bug Report

Description

After master restart with snapshot restore (enable_snapshot_restore=1), all memory segments lose their protocol field. The admin API returns "protocol": "" even though the segment was originally
mounted with "protocol": "tcp".

Additionally, OffsetBufferAllocator::replica_type_ is also lost during serialization, which affects NoF segment metric accounting (restored NoF segments are counted under "mem" instead of "nof").

Reproduction

Start Mooncake master with snapshot enabled
Mount a segment (e.g. protocol=tcp)
Wait for a snapshot to be generated
Restart the master
Query the admin API:

curl http://${MOONCAKE_MASTER_URL}/get_segments_detail | python3 -m json.tool

Output:

{
    "total_segments": 1,
    "segments": [
        {
            "segment_name": "xxx.xxx.xxx.148:13695",
            "segment_id": "6649975730718913833-14683276544698645121",
            "client_id": "9101436600682018864-15666146750897147799",
            "base_address": "0x773c56000000",
            "size_bytes": 3355443200,
            "size_human": "3.125 GiB",
            "te_endpoint": "xxx.xxx.xxx.148:13695",
            "protocol": "",        # ← empty after restore
            "status": "OK",
            "allocator_used_bytes": 3165446144,
            "allocator_capacity_bytes": 3355443200,
            "allocator_usage_percent": 94.34
        }
    ]
}

Note that protocol is empty even though the segment was mounted with --protocol tcp.

Root Cause

The snapshot serialization code in serializer.cpp uses custom msgpack serializers (not YLT_REFL auto-serialization) for both MountedSegment and OffsetBufferAllocator. These custom serializers missed two
fields:

MountedSegment::segment.protocol missing

Serialize (serializer.cpp:829-868) packs 8 elements:

segment.id, segment.name, segment.base, segment.size, segment.te_endpoint
status, has_buffer_allocator, buffer_allocator_data

But the Segment struct has 6 fields: id, name, base, size, te_endpoint, protocol. protocol is never serialized.

After restore, segment.protocol defaults to empty string "".

OffsetBufferAllocator::replica_type_ missing

Serialize (serializer.cpp:930-954) packs 6 elements:

segment_name, base, total_size, cur_size, transport_endpoint, offset_allocator

But OffsetBufferAllocator has replica_type_ which determines whether metrics are counted as MEMORY or NOF_SSD. replica_type_ is never serialized.

After restore, the allocator is reconstructed with the default ReplicaType::MEMORY, so any NoF segment's metrics are incorrectly counted under memory.

Fix Options

Option A: Add missing fields to serialization (no backward compat)

Add the fields at the natural position in the serialization arrays:

MountedSegment: add protocol after te_endpoint in both serialize/deserialize, change array size from 8→9
OffsetBufferAllocator: add replica_type_ after transport_endpoint_ in both serialize/deserialize, change array size from 6→7

Old snapshots become unreadable, but given that the restore cleanup bug (#TODO) already makes old snapshots unreliable, this is acceptable.

Option B: Backward-compatible fix (more complex)

Append the new fields at the end of the arrays and add size detection in deserialization:

Deserialize accepts both old (8) and new (9) element counts for MountedSegment
Deserialize accepts both old (6) and new (7) element counts for OffsetBufferAllocator

This preserves the ability to read old snapshots but adds complexity.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @mzygQAQ on GitHub (Jun 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2636 Originally assigned to: @mzygQAQ, @Icedcoco on GitHub. ### Bug Report ### Description After master restart with snapshot restore (`enable_snapshot_restore=1`), all memory segments lose their `protocol` field. The admin API returns `"protocol": ""` even though the segment was originally mounted with `"protocol": "tcp"`. Additionally, `OffsetBufferAllocator::replica_type_` is also lost during serialization, which affects NoF segment metric accounting (restored NoF segments are counted under "mem" instead of "nof"). ### Reproduction 1. Start Mooncake master with snapshot enabled 2. Mount a segment (e.g. protocol=tcp) 3. Wait for a snapshot to be generated 4. Restart the master 5. Query the admin API: ```bash curl http://${MOONCAKE_MASTER_URL}/get_segments_detail | python3 -m json.tool Output: { "total_segments": 1, "segments": [ { "segment_name": "xxx.xxx.xxx.148:13695", "segment_id": "6649975730718913833-14683276544698645121", "client_id": "9101436600682018864-15666146750897147799", "base_address": "0x773c56000000", "size_bytes": 3355443200, "size_human": "3.125 GiB", "te_endpoint": "xxx.xxx.xxx.148:13695", "protocol": "", # ← empty after restore "status": "OK", "allocator_used_bytes": 3165446144, "allocator_capacity_bytes": 3355443200, "allocator_usage_percent": 94.34 } ] } ``` Note that protocol is empty even though the segment was mounted with --protocol tcp. ### Root Cause The snapshot serialization code in serializer.cpp uses custom msgpack serializers (not YLT_REFL auto-serialization) for both MountedSegment and OffsetBufferAllocator. These custom serializers missed two fields: 1. MountedSegment::segment.protocol missing Serialize<MountedSegment> (serializer.cpp:829-868) packs 8 elements: - segment.id, segment.name, segment.base, segment.size, segment.te_endpoint - status, has_buffer_allocator, buffer_allocator_data But the Segment struct has 6 fields: id, name, base, size, te_endpoint, protocol. protocol is never serialized. After restore, segment.protocol defaults to empty string "". 2. OffsetBufferAllocator::replica_type_ missing Serialize<OffsetBufferAllocator> (serializer.cpp:930-954) packs 6 elements: - segment_name, base, total_size, cur_size, transport_endpoint, offset_allocator But OffsetBufferAllocator has replica_type_ which determines whether metrics are counted as MEMORY or NOF_SSD. replica_type_ is never serialized. After restore, the allocator is reconstructed with the default ReplicaType::MEMORY, so any NoF segment's metrics are incorrectly counted under memory. ### Fix Options Option A: Add missing fields to serialization (no backward compat) Add the fields at the natural position in the serialization arrays: - MountedSegment: add protocol after te_endpoint in both serialize/deserialize, change array size from 8→9 - OffsetBufferAllocator: add replica_type_ after transport_endpoint_ in both serialize/deserialize, change array size from 6→7 Old snapshots become unreadable, but given that the restore cleanup bug (#TODO) already makes old snapshots unreliable, this is acceptable. Option B: Backward-compatible fix (more complex) Append the new fields at the end of the arrays and add size detection in deserialization: - Deserialize accepts both old (8) and new (9) element counts for MountedSegment - Deserialize accepts both old (6) and new (7) element counts for OffsetBufferAllocator This preserves the ability to read old snapshots but adds complexity. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:15 +00:00

closed this issue
added the
stale

auto-closed

bug
labels

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 26, 2026):

Thanks for opening this issue, @mzygQAQ!

Field	Value
Issue	#2636
GitHub user ID	`30590325`
Reporter	@mzygQAQ

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 26, 2026): Thanks for opening this issue, @mzygQAQ! | Field | Value | |-------|-------| | **Issue** | #2636 | | **GitHub user ID** | `30590325` | | **Reporter** | @mzygQAQ | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@Icedcoco commented on GitHub (Jun 29, 2026):

Hi @mzygQAQ , I’ve completed the minimal fix for #2636 and opened PR #2668.

This update preserves MountedSegment.protocol and OffsetBufferAllocator.replica_type_ in snapshot serialization, while keeping legacy snapshot formats readable for backward compatibility. I did not extend the NoF service-level snapshot path in this change; the fix is intentionally scoped to the field-loss issue reported here.

Could you take a look at #2668 and let me know if the fix matches your expectations?

@Icedcoco commented on GitHub (Jun 29, 2026): Hi @mzygQAQ , I’ve completed the minimal fix for #2636 and opened PR #2668. This update preserves `MountedSegment.protocol` and `OffsetBufferAllocator.replica_type_` in snapshot serialization, while keeping legacy snapshot formats readable for backward compatibility. I did not extend the NoF service-level snapshot path in this change; the fix is intentionally scoped to the field-loss issue reported here. Could you take a look at #2668 and let me know if the fix matches your expectations?

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@mzygQAQ commented on GitHub (Jul 1, 2026):

Hi @mzygQAQ , I’ve completed the minimal fix for #2636 and opened PR #2668.

This update preserves MountedSegment.protocol and OffsetBufferAllocator.replica_type_ in snapshot serialization, while keeping legacy snapshot formats readable for backward compatibility. I did not extend the NoF service-level snapshot path in this change; the fix is intentionally scoped to the field-loss issue reported here.

Could you take a look at #2668 and let me know if the fix matches your expectations?

hi, @Icedcoco thanks for your work, I'm on a business trip and unable to verify it in the near future. I'm sorry — if you have the environment, you can go ahead and verify it yourself :)

@mzygQAQ commented on GitHub (Jul 1, 2026): > Hi [@mzygQAQ](https://github.com/mzygQAQ) , I’ve completed the minimal fix for [#2636](https://github.com/kvcache-ai/Mooncake/issues/2636) and opened PR [#2668](https://github.com/kvcache-ai/Mooncake/pull/2668). > > This update preserves `MountedSegment.protocol` and `OffsetBufferAllocator.replica_type_` in snapshot serialization, while keeping legacy snapshot formats readable for backward compatibility. I did not extend the NoF service-level snapshot path in this change; the fix is intentionally scoped to the field-loss issue reported here. > > Could you take a look at [#2668](https://github.com/kvcache-ai/Mooncake/pull/2668) and let me know if the fix matches your expectations? hi, @Icedcoco thanks for your work, I'm on a business trip and unable to verify it in the near future. I'm sorry — if you have the environment, you can go ahead and verify it yourself :)

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 30, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Sep 30, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:18:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 8, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:08 +00:00

[PR #707] [MERGED] docs: add troubleshooting steps for QP allocation error #1763