[GH-ISSUE #3140] [RFC]: Add Optional Object-Level CRC for Mooncake Store #806

New issue

Closed

opened 2026-10-29 21:19:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:19:44 +00:00

Owner

Originally created by @yokinoshitayoki on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3140

Changes proposed

Motivation

Mooncake Store currently relies on the underlying transfer and storage paths to preserve object data. When data corruption is observed, it can be difficult to determine whether the object changed before transfer, during transfer, or while stored.

This proposal adds an optional object-level CRC-64 checksum for diagnostic and testing purposes. When enabled, Mooncake Store records a checksum for the object written by the client and verifies that checksum after the object is read back. A mismatch is reported as an error, and the returned buffer must not be used.

The feature is intentionally opt-in because calculating a checksum requires scanning the complete object and may require device-to-host copies for GPU buffers.

Scope

The feature is enabled with:

MOONCAKE_STORE_CHECKSUM=1

When enabled:

put and upsert calculate a CRC-64/ECMA-182 checksum over the complete logical object before transfer.
Batch put and batch upsert preserve the association between each object key and its checksum.
Full-object get paths verify only the logical object bytes after the read completes.
CPU buffers are processed directly and incrementally.
GPU buffers are processed through bounded, chunked D2H staging.
The checksum is stored in Master object metadata and preserved in Master snapshots.
Local hot cache is bypassed so verification covers the underlying Store read path.
Objects without checksum metadata remain readable and are returned without verification.
A checksum mismatch fails closed with CHECKSUM_MISMATCH.

The implementation is contained within Mooncake Store and does not require changes to vLLM or SGLang.

Out of Scope

Partial or range-read verification.
Data repair, retry, or replica recovery after a mismatch.
Cryptographic integrity or authenticity guarantees.
Preserving checksum metadata through the operation log or guaranteeing checksum continuity across HA leader failover.
Avoiding the performance and memory-bandwidth overhead of full-object checksum calculation.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @yokinoshitayoki on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3140 ### Changes proposed ## Motivation Mooncake Store currently relies on the underlying transfer and storage paths to preserve object data. When data corruption is observed, it can be difficult to determine whether the object changed before transfer, during transfer, or while stored. This proposal adds an optional object-level CRC-64 checksum for diagnostic and testing purposes. When enabled, Mooncake Store records a checksum for the object written by the client and verifies that checksum after the object is read back. A mismatch is reported as an error, and the returned buffer must not be used. The feature is intentionally opt-in because calculating a checksum requires scanning the complete object and may require device-to-host copies for GPU buffers. ## Scope The feature is enabled with: ```bash MOONCAKE_STORE_CHECKSUM=1 ``` When enabled: - `put` and `upsert` calculate a CRC-64/ECMA-182 checksum over the complete logical object before transfer. - Batch `put` and batch `upsert` preserve the association between each object key and its checksum. - Full-object `get` paths verify only the logical object bytes after the read completes. - CPU buffers are processed directly and incrementally. - GPU buffers are processed through bounded, chunked D2H staging. - The checksum is stored in Master object metadata and preserved in Master snapshots. - Local hot cache is bypassed so verification covers the underlying Store read path. - Objects without checksum metadata remain readable and are returned without verification. - A checksum mismatch fails closed with `CHECKSUM_MISMATCH`. The implementation is contained within Mooncake Store and does not require changes to vLLM or SGLang. ## Out of Scope - Partial or range-read verification. - Data repair, retry, or replica recovery after a mismatch. - Cryptographic integrity or authenticity guarantees. - Preserving checksum metadata through the operation log or guaranteeing checksum continuity across HA leader failover. - Avoiding the performance and memory-bandwidth overhead of full-object checksum calculation. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:19:44 +00:00

closed this issue
added the
RFC
label

bsakaguchi commented

2026-10-29 21:19:45 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 27, 2026):

Thanks for opening this issue, @yokinoshitayoki!

Field	Value
Issue	#3140
GitHub user ID	`113957881`
Reporter	@yokinoshitayoki

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 27, 2026): Thanks for opening this issue, @yokinoshitayoki! | Field | Value | |-------|-------| | **Issue** | #3140 | | **GitHub user ID** | `113957881` | | **Reporter** | @yokinoshitayoki | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:30:44 +00:00

[PR #806] [MERGED] [Chores] Offset Allocator Test Fix & Docs Fix #1835