[PR #255] [MERGED] Fix Hybrid upper growth during discard/rewrite cycles #296

New issue

Closed

opened 2026-10-29 14:21:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/255
Author: @huang-jl
Created: 9/7/2026
Status: ✅ Merged
Merged: 9/10/2026
Merged by: @yingdi-shan

Base: main ← Head: hjl/overlaybd-write-after-discard

📝 Commits (2)

3259815 refactor(lsmt): distinguish backed zero mappings
a086936 fix(lsmt): reuse hybrid extents after discard

📊 Changes

6 files changed (+999 additions, -85 deletions)

View changed files

📝 storage/overlaybd/src/lsmt/file/helper.rs (+65 -4)
📝 storage/overlaybd/src/lsmt/file/readwrite.rs (+123 -28)
📝 storage/overlaybd/src/lsmt/file/tests.rs (+633 -27)
📝 storage/overlaybd/src/lsmt/file/types.rs (+8 -0)
📝 storage/overlaybd/src/lsmt/format.rs (+18 -0)
📝 storage/overlaybd/src/lsmt/index.rs (+152 -26)

📄 Description

What

Preserve physical allocations across Hybrid discards so subsequent writes can reuse existing space, while keeping discard index-only.

Why

Previously, discard replaced a data mapping with a zero mapping without retaining its physical address. Rewriting that range then appended new data, leaving the previous allocation unreachable.

Repeated discard/rewrite cycles could therefore grow upper.data beyond the virtual disk size.

Scope and non-goals

Only update the behavior of hybrid read write (upper).

Design and behavior changes

Discard still returns zeros and masks lower-layer data without writing zeros or punching holes.
Rewrites reuse allocated zero ranges; only unallocated ranges append data.
Zero-to-data transitions still append index records for persistence.

Compatibility and operations

Existing read-only lower layers remain compatible. Pre-change writable uppers are not supported; deployment requires creating fresh uppers.
Sparse and pure LogStructured I/O behavior remains unchanged.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/255 **Author:** [@huang-jl](https://github.com/huang-jl) **Created:** 9/7/2026 **Status:** ✅ Merged **Merged:** 9/10/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `hjl/overlaybd-write-after-discard` --- ### 📝 Commits (2) - [`3259815`](https://github.com/kvcache-ai/AgentENV/commit/325981507dbe7db67047e86d46dad3a45040509d) refactor(lsmt): distinguish backed zero mappings - [`a086936`](https://github.com/kvcache-ai/AgentENV/commit/a086936b387288b8dc6e161d02af6a6a9813929f) fix(lsmt): reuse hybrid extents after discard ### 📊 Changes **6 files changed** (+999 additions, -85 deletions) <details> <summary>View changed files</summary> 📝 `storage/overlaybd/src/lsmt/file/helper.rs` (+65 -4) 📝 `storage/overlaybd/src/lsmt/file/readwrite.rs` (+123 -28) 📝 `storage/overlaybd/src/lsmt/file/tests.rs` (+633 -27) 📝 `storage/overlaybd/src/lsmt/file/types.rs` (+8 -0) 📝 `storage/overlaybd/src/lsmt/format.rs` (+18 -0) 📝 `storage/overlaybd/src/lsmt/index.rs` (+152 -26) </details> ### 📄 Description ## What Preserve physical allocations across Hybrid discards so subsequent writes can reuse existing space, while keeping discard index-only. ## Why Previously, discard replaced a data mapping with a zero mapping without retaining its physical address. Rewriting that range then appended new data, leaving the previous allocation unreachable. Repeated discard/rewrite cycles could therefore grow upper.data beyond the virtual disk size. ## Scope and non-goals Only update the behavior of hybrid read write (upper). ## Design and behavior changes - Discard still returns zeros and masks lower-layer data without writing zeros or punching holes. - Rewrites reuse allocated zero ranges; only unallocated ranges append data. - Zero-to-data transitions still append index records for persistence. ## Compatibility and operations - Existing read-only lower layers remain compatible. Pre-change writable uppers are not supported; deployment requires creating fresh uppers. - Sparse and pure LogStructured I/O behavior remains unchanged. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [ ] Documentation updated - [ ] Benchmarks or performance comparison completed --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:55 +00:00