[PR #1381] [Store]: add ETCD snapshot object store backend in HA mode #2208

New issue

Open

opened 2026-10-29 21:33:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:33:40 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1381
Author: @00fish0
Created: 1/14/2026
Status: 🔄 Open

Base: main ← Head: dev/snapshot_etcd_storage

📝 Commits (2)

110dfe1 snapshot etcd backend
9445d61 fix ai review feedback

📊 Changes

19 files changed (+1145 additions, -49 deletions)

View changed files

📝 mooncake-common/etcd/etcd_wrapper.go (+38 -0)
➕ mooncake-store/include/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store.h (+62 -0)
📝 mooncake-store/include/ha/snapshot/object/snapshot_object_store.h (+30 -7)
➕ mooncake-store/include/ha/snapshot/snapshot_socket_util.h (+48 -0)
📝 mooncake-store/include/master_config.h (+14 -1)
📝 mooncake-store/include/master_service.h (+11 -0)
📝 mooncake-store/src/CMakeLists.txt (+1 -0)
📝 mooncake-store/src/ha/snapshot/catalog_backed_snapshot_provider.cpp (+2 -1)
➕ mooncake-store/src/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store.cpp (+176 -0)
📝 mooncake-store/src/ha/snapshot/object/snapshot_object_store.cpp (+15 -2)
📝 mooncake-store/src/master.cpp (+1 -1)
📝 mooncake-store/src/master_service.cpp (+381 -24)
📝 mooncake-store/tests/CMakeLists.txt (+7 -0)
📝 mooncake-store/tests/ha/snapshot/catalog_backed_snapshot_provider_test.cpp (+23 -5)
📝 mooncake-store/tests/ha/snapshot/master_service_test_for_snapshot_base.h (+1 -1)
➕ mooncake-store/tests/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store_test.cpp (+180 -0)
➕ mooncake-store/tests/ha/snapshot/snapshot_socket_util_test.cpp (+97 -0)
📝 mooncake-store/tests/ha/snapshot/snapshot_test_utils.h (+37 -2)
📝 mooncake-store/tests/ha/standby/hot_standby_snapshot_bootstrap_test.cpp (+21 -5)

📄 Description

Description

Add ETCD as a snapshot object store backend, enabling master state snapshots to be persisted directly to the etcd cluster already used for HA coordination — no additional storage infrastructure required.

Key design decisions

socketpair-based IPC: Using socketpair() between the forked child (serialization) and parent process (upload). This solves the core problem: the Go runtime (cgo) cannot be called from a fork()-ed child process.
Reuse EmbeddedSnapshotCatalogStore: Instead of creating a dedicated EtcdSnapshotCatalogStore, the embedded catalog stores its index data (latest.txt, descriptor.txt) as regular keys via the SnapshotObjectStore interface. With EtcdSnapshotObjectStore, these keys are automatically stored in etcd.
Per-component ack for memory control: The child serializes and sends one component at a time (metadata → segments → task_manager → manifest), waiting for the parent's ack before proceeding to the next. This keeps peak memory to max(single component size) in both processes.

Architecture: ETCD vs Local/S3

	Local / S3	ETCD
Child process	serialize + upload	serialize only → write to socketpair
Parent process	waitpid (passive)	read socketpair → upload via cgo → publish catalog
Why	Child can call object store directly	Child cannot call cgo after fork (Go runtime corrupted)
Restore	Same for all backends — main process calls SnapshotObjectStore::DownloadBuffer()

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

Unit tests: etcd_snapshot_object_store_test (10 cases), snapshot_socket_util_test
E2E: persist → SIGKILL → restore → metadata intact → surviving client get() data byte-for-byte match

Checklist

I have performed a self-review of my own code.
I have formatted my own code using ./scripts/code_format.sh before submitting.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1381 **Author:** [@00fish0](https://github.com/00fish0) **Created:** 1/14/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `dev/snapshot_etcd_storage` --- ### 📝 Commits (2) - [`110dfe1`](https://github.com/kvcache-ai/Mooncake/commit/110dfe18541241c3ce8807ae7dbee4624fb933b1) snapshot etcd backend - [`9445d61`](https://github.com/kvcache-ai/Mooncake/commit/9445d61bdd5cfbe4b40cb7b1c7ac4e19733065ee) fix ai review feedback ### 📊 Changes **19 files changed** (+1145 additions, -49 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-common/etcd/etcd_wrapper.go` (+38 -0) ➕ `mooncake-store/include/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store.h` (+62 -0) 📝 `mooncake-store/include/ha/snapshot/object/snapshot_object_store.h` (+30 -7) ➕ `mooncake-store/include/ha/snapshot/snapshot_socket_util.h` (+48 -0) 📝 `mooncake-store/include/master_config.h` (+14 -1) 📝 `mooncake-store/include/master_service.h` (+11 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+1 -0) 📝 `mooncake-store/src/ha/snapshot/catalog_backed_snapshot_provider.cpp` (+2 -1) ➕ `mooncake-store/src/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store.cpp` (+176 -0) 📝 `mooncake-store/src/ha/snapshot/object/snapshot_object_store.cpp` (+15 -2) 📝 `mooncake-store/src/master.cpp` (+1 -1) 📝 `mooncake-store/src/master_service.cpp` (+381 -24) 📝 `mooncake-store/tests/CMakeLists.txt` (+7 -0) 📝 `mooncake-store/tests/ha/snapshot/catalog_backed_snapshot_provider_test.cpp` (+23 -5) 📝 `mooncake-store/tests/ha/snapshot/master_service_test_for_snapshot_base.h` (+1 -1) ➕ `mooncake-store/tests/ha/snapshot/object/backends/etcd/etcd_snapshot_object_store_test.cpp` (+180 -0) ➕ `mooncake-store/tests/ha/snapshot/snapshot_socket_util_test.cpp` (+97 -0) 📝 `mooncake-store/tests/ha/snapshot/snapshot_test_utils.h` (+37 -2) 📝 `mooncake-store/tests/ha/standby/hot_standby_snapshot_bootstrap_test.cpp` (+21 -5) </details> ### 📄 Description ## Description Add ETCD as a snapshot object store backend, enabling master state snapshots to be persisted directly to the etcd cluster already used for HA coordination — no additional storage infrastructure required. ### Key design decisions 1. socketpair-based IPC: Using socketpair() between the forked child (serialization) and parent process (upload). This solves the core problem: the Go runtime (cgo) cannot be called from a fork()-ed child process. 2. Reuse EmbeddedSnapshotCatalogStore: Instead of creating a dedicated EtcdSnapshotCatalogStore, the embedded catalog stores its index data (latest.txt, descriptor.txt) as regular keys via the SnapshotObjectStore interface. With EtcdSnapshotObjectStore, these keys are automatically stored in etcd. 3. Per-component ack for memory control: The child serializes and sends one component at a time (metadata → segments → task_manager → manifest), waiting for the parent's ack before proceeding to the next. This keeps peak memory to max(single component size) in both processes. Architecture: ETCD vs Local/S3 | Local / S3 | ETCD -- | -- | -- Child process | serialize + upload | serialize only → write to socketpair Parent process | waitpid (passive) | read socketpair → upload via cgo → publish catalog Why | Child can call object store directly | Child cannot call cgo after fork (Go runtime corrupted) Restore | Same for all backends — main process calls SnapshotObjectStore::DownloadBuffer() | ## Type of Change * Types - [ ] Bug fix - [x] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? 1. Unit tests: etcd_snapshot_object_store_test (10 cases), snapshot_socket_util_test 2. E2E: persist → SIGKILL → restore → metadata intact → surviving client get() data byte-for-byte match ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have formatted my own code using `./scripts/code_format.sh` before submitting. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:33:40 +00:00