[GH-ISSUE #4346] [RFC]: Opt-in client-side FP8 codec for complete Store KV chunks #1091

New issue

Open

opened 2026-10-29 21:22:47 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:22:47 +00:00

Owner

Originally created by @stmatengss on GitHub (Sep 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4346

Changes proposed

Motivation and scope

Applications that store FP16/BF16 KV chunks in Mooncake currently supply bytes in their chosen representation. An application opting into lossy storage needs an explicit contract for quantization, scales, reconstruction, format compatibility and failure behavior. Changing only a tensor dtype does not define that storage contract.

Propose an optional codec above the existing Store bytes API, with a CPU scaled-E4M3FN implementation as the first concrete algorithm. This reduces the stored representation size for sufficiently large chunks; it does not make attention execute in FP8, and it does not establish a throughput or model-quality improvement.

This RFC accompanies a draft implementation for review. The proposed v1 format should not be treated as a released compatibility promise until reviewed.

Design: three responsibilities

write: CPU FP16/BF16 tensor -> KVCodec -> complete encoded record -> Store.put
read:  Store.get -> complete encoded record -> validate -> KVCodec -> caller tensor

KVCodec: synchronous C++ buffer transforms, descriptor validation, exact encoded size and actual bytes written. ScaledFp8Codec is the first implementation. The core has no Torch, Store RPC, CUDA or RDMA dependency.
Encoded record: one self-describing byte sequence containing the version, original dtype and shape, group size, element count, FP32 scales, FP8 payload and whole-record CRC32. This is a wire contract, not a separate object hierarchy.
CodecStoreClient: a thin Python facade over an initialized MooncakeDistributedStore, with put, get_into and storage_key. It handles tensor validation, representation keys and synchronous buffer lifetime.

The master and Transfer Engine continue storing/transferring opaque bytes. Framework adapters still own model/token/layout/shard identity, K/V interpretation, page tables and component completeness. TensorDesc intentionally describes only contiguous CPU memory; it does not replace the existing reshard placement descriptors.

Numeric and storage contract

Accept finite, nonempty, contiguous CPU FP16/BF16 tensors of rank 1–8. Reject packed FP8 inputs rather than quantizing them again.
Group consecutive flattened elements, with a short final group allowed. There is no implicit per-head or per-token axis interpretation.
Persist one FP32 scale per group: max(abs(x))/448, or 1 for an all-zero group. Normalize in FP32, clamp and convert to E4M3FN with round-to-nearest-even.
Reconstruct into the original dtype, clamp to its finite range, and preserve signed zero. This is lossy reconstruction.
Use a single little-endian record. Its CRC32/IEEE covers both metadata and payload; it detects accidental corruption, not adversarial modification.
Namespace records as __mc_codec__/fp8-e4m3fn-v1-g<group_size>/<logical-key>. The logical key must already identify the model revision, token sequence, original dtype/layout and shard. The prefix isolates representations, not tenants or model identity.
Read the complete object once; check version, checksum, lengths, descriptor and side data before modifying the destination. Preserve Store put status, insert-only semantics and replica configuration.
An empty bytes get is reported as StoreReadError, because that API does not distinguish absence from read failure.

For N elements and rank R, the record is 32 + 8*R + 4*ceil(N/group_size) + N bytes, versus 2*N raw. At group size 128, the asymptotic ratio is about 1.94x before the fixed header and Store overhead. Small chunks or groups can expand. No adaptive raw fallback is proposed.

Relation to LMCache and existing Mooncake work

LMCache's distributed serde separates synchronous serialization, asynchronous execution and the L2 storage wrapper. We borrow the transform/backend boundary; this first implementation keeps execution synchronous to avoid introducing thread pools, futures and cancellation semantics before they are needed. The inspected LMCache FP8 serde performs a direct FP8 cast. This proposal adds group scales and an independently validated record; it is not wire-compatible with LMCache serde.

Related work was checked before filing:

#1673 proposes object-type classification and type-aware Store policies. This proposal concerns numerical representation and does not add master classification or policy machinery.
#3564 adds KV reshard placement/planning and explicitly excludes quantization conversion. This codec does not perform resharding or change those descriptor semantics.
#3534 unifies Python packaging. The implementation uses the canonical python/mooncake source and private native extension, with staging for the existing legacy wheel builder.
Existing structured-object codecs serialize Python object structure; this proposal supplies a narrowly scoped lossy tensor transformation.

Searches of issues and open PRs for quantization, FP8, codec and KV compression found no existing proposal implementing this same contract.

Alternatives and deliberate exclusions

Quantize in the master or TE: would introduce model/numerical semantics into byte transport and storage. Keep the transform at the client boundary.
Extend reshard dtype metadata alone: would not specify scales, rounding, framing or reconstruction; it also couples storage to transfer planning.
Full plugin registry, execution plans and asynchronous executors now: defer until there is a second algorithm/device or concrete async integration. Do not create unused abstractions.
Raw fallback under the same key: reject unsupported inputs explicitly; fallback would make compatibility and failure interpretation ambiguous.

GPU/NPU kernels, batch/range reads, asynchronous execution, multi-component transactions, framework serving integration and new eviction/replication policies are outside the initial scope.

Validation and acceptance

The draft implements independent PyTorch reference comparisons, exhaustive finite FP16/BF16 input-bit-pattern coverage, a golden wire fixture, corruption/configuration mismatch checks and Store-error tests. C++ buffer tests cover unaligned memory, capacity, overlap, every truncated length and every single-bit corruption, including ASan/UBSan.

Before serving adoption: validate a real Linux Store TCP round trip, then relevant RDMA/GPU/framework paths; measure encode/decode time, temporary memory, persisted and transferred bytes, retrieval latency, and model quality on a fixed workload. The CPU implementation is a reference baseline, not a bandwidth-saturating codec.

Review questions: Is the flattened-group contract an appropriate first boundary? Should v1 framing/namespace be accepted as specified? Which workload and quality threshold should gate the first serving adapter?

Before submitting a new issue...

Searched related issues and pull requests and reviewed the contribution guidelines and Store documentation.

AI assistance: Codex assisted with source research, implementation, tests and this proposal. Human line-by-line review remains pending; the associated implementation is a draft.

Originally created by @stmatengss on GitHub (Sep 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4346 ### Changes proposed ## Motivation and scope Applications that store FP16/BF16 KV chunks in Mooncake currently supply bytes in their chosen representation. An application opting into lossy storage needs an explicit contract for quantization, scales, reconstruction, format compatibility and failure behavior. Changing only a tensor dtype does not define that storage contract. Propose an optional codec above the existing Store bytes API, with a CPU scaled-E4M3FN implementation as the first concrete algorithm. This reduces the stored representation size for sufficiently large chunks; it does not make attention execute in FP8, and it does not establish a throughput or model-quality improvement. This RFC accompanies a draft implementation for review. The proposed v1 format should not be treated as a released compatibility promise until reviewed. ## Design: three responsibilities ```text write: CPU FP16/BF16 tensor -> KVCodec -> complete encoded record -> Store.put read: Store.get -> complete encoded record -> validate -> KVCodec -> caller tensor ``` 1. **KVCodec**: synchronous C++ buffer transforms, descriptor validation, exact encoded size and actual bytes written. `ScaledFp8Codec` is the first implementation. The core has no Torch, Store RPC, CUDA or RDMA dependency. 2. **Encoded record**: one self-describing byte sequence containing the version, original dtype and shape, group size, element count, FP32 scales, FP8 payload and whole-record CRC32. This is a wire contract, not a separate object hierarchy. 3. **CodecStoreClient**: a thin Python facade over an initialized `MooncakeDistributedStore`, with `put`, `get_into` and `storage_key`. It handles tensor validation, representation keys and synchronous buffer lifetime. The master and Transfer Engine continue storing/transferring opaque bytes. Framework adapters still own model/token/layout/shard identity, K/V interpretation, page tables and component completeness. `TensorDesc` intentionally describes only contiguous CPU memory; it does not replace the existing reshard placement descriptors. ## Numeric and storage contract - Accept finite, nonempty, contiguous CPU FP16/BF16 tensors of rank 1–8. Reject packed FP8 inputs rather than quantizing them again. - Group consecutive flattened elements, with a short final group allowed. There is no implicit per-head or per-token axis interpretation. - Persist one FP32 scale per group: `max(abs(x))/448`, or 1 for an all-zero group. Normalize in FP32, clamp and convert to E4M3FN with round-to-nearest-even. - Reconstruct into the original dtype, clamp to its finite range, and preserve signed zero. This is lossy reconstruction. - Use a single little-endian record. Its CRC32/IEEE covers both metadata and payload; it detects accidental corruption, not adversarial modification. - Namespace records as `__mc_codec__/fp8-e4m3fn-v1-g<group_size>/<logical-key>`. The logical key must already identify the model revision, token sequence, original dtype/layout and shard. The prefix isolates representations, not tenants or model identity. - Read the complete object once; check version, checksum, lengths, descriptor and side data before modifying the destination. Preserve Store put status, insert-only semantics and replica configuration. - An empty bytes `get` is reported as `StoreReadError`, because that API does not distinguish absence from read failure. For `N` elements and rank `R`, the record is `32 + 8*R + 4*ceil(N/group_size) + N` bytes, versus `2*N` raw. At group size 128, the asymptotic ratio is about 1.94x before the fixed header and Store overhead. Small chunks or groups can expand. No adaptive raw fallback is proposed. ## Relation to LMCache and existing Mooncake work LMCache's distributed serde separates synchronous serialization, asynchronous execution and the L2 storage wrapper. We borrow the transform/backend boundary; this first implementation keeps execution synchronous to avoid introducing thread pools, futures and cancellation semantics before they are needed. The inspected [LMCache FP8 serde](https://github.com/LMCache/LMCache/blob/05fc77a0a7ababd9a7f2e343a771bc4bbc5b65cb/lmcache/v1/distributed/serde/fp8.py) performs a direct FP8 cast. This proposal adds group scales and an independently validated record; it is not wire-compatible with LMCache serde. Related work was checked before filing: - #1673 proposes object-type classification and type-aware Store policies. This proposal concerns numerical representation and does not add master classification or policy machinery. - #3564 adds KV reshard placement/planning and explicitly excludes quantization conversion. This codec does not perform resharding or change those descriptor semantics. - #3534 unifies Python packaging. The implementation uses the canonical `python/mooncake` source and private native extension, with staging for the existing legacy wheel builder. - Existing structured-object codecs serialize Python object structure; this proposal supplies a narrowly scoped lossy tensor transformation. Searches of issues and open PRs for quantization, FP8, codec and KV compression found no existing proposal implementing this same contract. ## Alternatives and deliberate exclusions - **Quantize in the master or TE:** would introduce model/numerical semantics into byte transport and storage. Keep the transform at the client boundary. - **Extend reshard dtype metadata alone:** would not specify scales, rounding, framing or reconstruction; it also couples storage to transfer planning. - **Full plugin registry, execution plans and asynchronous executors now:** defer until there is a second algorithm/device or concrete async integration. Do not create unused abstractions. - **Raw fallback under the same key:** reject unsupported inputs explicitly; fallback would make compatibility and failure interpretation ambiguous. GPU/NPU kernels, batch/range reads, asynchronous execution, multi-component transactions, framework serving integration and new eviction/replication policies are outside the initial scope. ## Validation and acceptance The draft implements independent PyTorch reference comparisons, exhaustive finite FP16/BF16 input-bit-pattern coverage, a golden wire fixture, corruption/configuration mismatch checks and Store-error tests. C++ buffer tests cover unaligned memory, capacity, overlap, every truncated length and every single-bit corruption, including ASan/UBSan. Before serving adoption: validate a real Linux Store TCP round trip, then relevant RDMA/GPU/framework paths; measure encode/decode time, temporary memory, persisted and transferred bytes, retrieval latency, and model quality on a fixed workload. The CPU implementation is a reference baseline, not a bandwidth-saturating codec. Review questions: Is the flattened-group contract an appropriate first boundary? Should v1 framing/namespace be accepted as specified? Which workload and quality threshold should gate the first serving adapter? ### Before submitting a new issue... - [x] Searched related issues and pull requests and reviewed the contribution guidelines and Store documentation. AI assistance: Codex assisted with source research, implementation, tests and this proposal. Human line-by-line review remains pending; the associated implementation is a draft.

bsakaguchi added the

RFC

label

2026-10-29 21:22:47 +00:00

bsakaguchi commented

2026-10-29 21:22:48 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 27, 2026):

Thanks for opening this issue, @stmatengss!

Field	Value
Issue	#4346
GitHub user ID	`11641725`
Reporter	@stmatengss

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 27, 2026): Thanks for opening this issue, @stmatengss! | Field | Value | |-------|-------| | **Issue** | #4346 | | **GitHub user ID** | `11641725` | | **Reporter** | @stmatengss | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.