kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4856] [Bug]: [Store] Standalone master spends 20–48 s destroying its index after SIGTERM (12M–25M objects) #1235

New issue

Open

opened 2026-10-29 21:24:15 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:24:15 +00:00

Owner

Originally created by @Juhyun-Kim-Memphis on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4856

Bug Report

Summary

On SIGTERM, a standalone (non-HA) mooncake_master stops its RPC server within about 0.1 s and returns from main(). The destructors then tear down the object index one object at a time on the main thread. With 12M objects this took 20.7 s from the signal to process exit, and with 25M objects 48.2 s (median of 3 runs each, current main). More than 90% of that time is spent destroying the index. For comparison, after SIGKILL the kernel needed 1.1 s (12M) and 2.4 s (25M) to tear down the same process.

The master stops serving RPCs at the start of this window. A standalone master does not keep its index across a restart (the legacy snapshot was removed in #4406), so the result of this work is thrown away. On Kubernetes, the pod stays in Terminating for this time, and a Recreate rollout starts the new master only after the old pod is gone, so the time adds to the period without a master. At 25M objects all three runs also took longer than the default terminationGracePeriodSeconds (30 s), so in a pod with the default the kubelet would end the exit with SIGKILL.

The HA branch does not run this teardown: since #4745, a shutdown ends the process with _exit after the drain, before the service is destroyed.

Related: #4666 (the planned-restart RFC; its appendix listed this item as a separate report).

Where in the code

Line numbers are for main at 8c19b728 (2026-10-22), the commit we measured.

The non-HA branch waits for the signal flag, stops the RPC server and returns 0 from main(). The objects on the stack, including the WrappedMasterService that owns the MasterService, are destroyed on the way out:

github.com/kvcache-ai/Mooncake@8c19b7286c/mooncake-store/src/master.cpp (L1742-L1766)
~MasterService stops and joins its threads (master_service.cpp L550-L612). After that, its members are destroyed, including metadata_shards_: 1,024 shards, each an unordered_map<TenantId, TenantState> whose TenantState holds an unordered_map<std::string, ObjectRecord>:

github.com/kvcache-ai/Mooncake@8c19b7286c/mooncake-store/include/master_service.h (L1101-L1136)
For every object this destroys the key, the ObjectMetadata (its destructor decrements the key-count gauge, object_metadata.h L61-L66), the vector<ObjectPart> (object_metadata.h L503), each part's vector<Replica>, and for a memory replica the AllocatedBuffer and its OffsetAllocationHandle. Each of those two locks a weak_ptr to its allocator (allocator.cpp L66-L75, offset_allocator.cpp L531-L536). Neither OffsetBufferAllocator::deallocate nor OffsetAllocator::freeAllocation appears in our profile. We think the allocators are already gone at that point: segment_manager_ is declared after metadata_shards_ (master_service.h L2328), so it is destroyed first. If so, the time goes to walking and freeing the nodes, not to returning space to the allocators.
For comparison, in the HA branch a shutdown ends the process in release_service() before wrapped_master_service.reset(), through shutdown_exit and ExitAfterShutdown (_exit, "no destructors run"):

github.com/kvcache-ai/Mooncake@8c19b7286c/mooncake-store/src/ha/leadership/master_service_supervisor.cpp (L733-L742)

(shutdown_exit is at L437-L444, ExitAfterShutdown in include/shutdown_request.h L39-L42.)

The non-HA signal handling came from #2359. That PR's goal was to stop the RPC server and the HTTP metadata server cleanly and to join the metrics thread. It did not discuss the index teardown.

Observed

Median of 3 runs; the range is in parentheses. Times are seconds after the signal.

	12M objects	25M objects
Master RSS before the signal	14.5 GiB	30.2 GiB
SIGTERM to process exit	20.7 (20.1–20.9)	48.2 (44.4–50.5)
Inside it: RPC server stopped (log line `Shutdown signal received`)	0.01–0.08	0.01–0.07
Inside it: `~MasterService` threads joined (log line `Client offboarding worker stopped`)	0.84–0.88	0.11–0.19
Inside it: from there until `HTTP metadata server stopped`, which runs after the service is destroyed	18.8 (18.3–19.0)	46.0 (42.1–48.4)
Inside it: from there until exit (static destructors and the kernel's teardown)	1.05–1.07	1.94–2.13
Control: SIGKILL to process exit	1.07 (0.99–1.11)	2.37 (2.32–2.40)
Master pinned to one NUMA node, SIGTERM to exit (1 run)	13.0	25.9

The exiting process uses one core the whole time. In the 12M run 1 it used 13.5 s of user time and 5.8 s of system time. In the 25M run 1 it used 28.3 s and 15.8 s. Pinned to one NUMA node (numactl --cpunodebind=0 --membind=0), system time dropped to 0.6 s (12M) and 1.2 s (25M). The user time stayed at about 1 µs per object.

Profile (12M objects; perf record --call-graph dwarf at 499 Hz from 1 s before the signal to exit). The mooncake_master target is linked with -s, so for this run we relinked the same object files without -s; the .text section is identical. 89.5% of the process's on-CPU samples are under:

main
 -> ~shared_ptr<WrappedMasterService>
  -> MasterService::~MasterService()
   -> std::array<MasterService::MetadataShard, 1024>::~array()
    -> unordered_map<TenantId, TenantState>  ->  unordered_map<std::string, ObjectRecord>::~unordered_map()
     -> ~ObjectRecord -> ~ObjectMetadata -> ~vector<ObjectPart> -> ~vector<Replica>
      -> ~AllocatedBuffer / ~OffsetAllocationHandle (weak_ptr lock and release), free()

Self time: the hash-node deallocation loop 34.0%, ~AllocatedBuffer 7.0%, shared_ptr control-block release 6.9%, libc (free and unresolved internals) 17.7%, ~OffsetAllocationHandle 3.0%, the key-count gauge 0.6%. The kernel had 30% of all samples: 24% in page faults, of which 15% are automatic NUMA balancing faults (do_numa_page, with 12.5% in migrate_misplaced_folio) as the single exiting thread walks memory that the RPC threads allocated on both sockets, and 5% in the address-space teardown at exit (exit_mmap).

For reference, we had measured 21.9 s for the same graceful exit on an older build based on upstream 245e7106, with 12.17M objects (most with a local-disk replica, 7.0M of them disk-only), real stores and a 4-CPU limit. The setups differ, but the current main is not faster.

Setup

main at 8c19b728, CMAKE_BUILD_TYPE=Release (-O3 -DNDEBUG), -DUSE_HTTP=ON, no etcd, default glibc malloc.
Master flags: --rpc_port=50051 --metrics_port=9003 --enable_http_metadata_server=true --http_metadata_server_host=0.0.0.0 --http_metadata_server_port=8080 --client_ttl=60 -enable_offload=true. All other flags were defaults (standalone, offset allocator, 16 RPC threads).
Index: 12 fake segments × 675 GiB, mounted through MountSegment by a test client that keeps sending Ping until the end (metadata only, no data, no transfer engine). Objects are put with BatchPutStart/BatchPutEnd: 256 KiB each, 96-byte keys, one memory replica, no disk replicas. 12.0M and 25.0M objects use 36% and 75% of the capacity, so no eviction ran. We used the load tool attached to #4818 (--mode=segments, then --mode=fill --threads=48 --put_batch=64 --value_size=262144). Filling took 35–40 s (12M) and 80–100 s (25M).
Host: 2 × Intel Xeon Platinum 8462Y+ (128 logical CPUs, 2 NUMA nodes), Linux 6.8, kernel.numa_balancing=1. The master ran in a container that was not CPU-limited and was not pinned, except in the two pinned runs.
We measured from the signal to process exit with the shell's wait on the master's PID. The shell was the master's parent. Kubernetes was not in the loop, so kubelet and runtime overheads are not included.
5 s idle between the end of the fill and the signal.

Limits

One host type. The unpinned numbers depend on the host's NUMA layout and on NUMA balancing; the pinned case (about 1 µs per object) is closer to a single-socket host. We ran the pinned case once per size.
The 25M runs grew from run to run (44.4, 48.2, 50.5 s, and 55.3 s in a later run under perf). We did not find the cause. The 12M runs stayed within 0.8 s of each other.
Only memory replicas, one per object. Disk replicas and multi-replica objects add destructor work per replica that we did not measure on this build.
We did not try other allocators (jemalloc, tcmalloc) or glibc tunables.

Possible directions (suggestions only)

In the non-HA branch, end the process without destroying the index once the RPC server and the service threads have stopped, as the HA branch does since #4745. The explicit steps from #2359 (stop the RPC server and the HTTP metadata server, join the threads) and a log flush could run first, then _exit(0). With the measurements above, the exit would then take about as long as the thread joins plus the SIGKILL time (about 2–2.5 s in these runs). We have not checked whether any destructor further down has an effect outside the process that would need to run first.
A narrower variant: on the shutdown path only, let MasterService leave metadata_shards_ undestroyed (for example, move the maps into a heap object that is intentionally not freed before process exit), and keep the rest of the destructors.
Destroying the 1,024 shards in parallel would shorten the wall time but keep the work. We list it only for completeness.

Before submitting...

Ensure you searched for relevant issues and read the documentation

Originally created by @Juhyun-Kim-Memphis on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4856 ### Bug Report #### Summary On SIGTERM, a standalone (non-HA) `mooncake_master` stops its RPC server within about 0.1 s and returns from `main()`. The destructors then tear down the object index one object at a time on the main thread. With 12M objects this took **20.7 s** from the signal to process exit, and with 25M objects **48.2 s** (median of 3 runs each, current `main`). More than 90% of that time is spent destroying the index. For comparison, after SIGKILL the kernel needed 1.1 s (12M) and 2.4 s (25M) to tear down the same process. The master stops serving RPCs at the start of this window. A standalone master does not keep its index across a restart (the legacy snapshot was removed in #4406), so the result of this work is thrown away. On Kubernetes, the pod stays in Terminating for this time, and a `Recreate` rollout starts the new master only after the old pod is gone, so the time adds to the period without a master. At 25M objects all three runs also took longer than the default `terminationGracePeriodSeconds` (30 s), so in a pod with the default the kubelet would end the exit with SIGKILL. The HA branch does not run this teardown: since #4745, a shutdown ends the process with `_exit` after the drain, before the service is destroyed. Related: #4666 (the planned-restart RFC; its appendix listed this item as a separate report). #### Where in the code Line numbers are for `main` at `8c19b728` (2026-10-22), the commit we measured. 1. The non-HA branch waits for the signal flag, stops the RPC server and returns 0 from `main()`. The objects on the stack, including the `WrappedMasterService` that owns the `MasterService`, are destroyed on the way out: https://github.com/kvcache-ai/Mooncake/blob/8c19b7286c5ef46a5f2bf848f93acd86439a1b4e/mooncake-store/src/master.cpp#L1742-L1766 2. `~MasterService` stops and joins its threads (`master_service.cpp` L550-L612). After that, its members are destroyed, including `metadata_shards_`: 1,024 shards, each an `unordered_map<TenantId, TenantState>` whose `TenantState` holds an `unordered_map<std::string, ObjectRecord>`: https://github.com/kvcache-ai/Mooncake/blob/8c19b7286c5ef46a5f2bf848f93acd86439a1b4e/mooncake-store/include/master_service.h#L1101-L1136 3. For every object this destroys the key, the `ObjectMetadata` (its destructor decrements the key-count gauge, `object_metadata.h` L61-L66), the `vector<ObjectPart>` (`object_metadata.h` L503), each part's `vector<Replica>`, and for a memory replica the `AllocatedBuffer` and its `OffsetAllocationHandle`. Each of those two locks a `weak_ptr` to its allocator (`allocator.cpp` L66-L75, `offset_allocator.cpp` L531-L536). Neither `OffsetBufferAllocator::deallocate` nor `OffsetAllocator::freeAllocation` appears in our profile. We think the allocators are already gone at that point: `segment_manager_` is declared after `metadata_shards_` (`master_service.h` L2328), so it is destroyed first. If so, the time goes to walking and freeing the nodes, not to returning space to the allocators. 4. For comparison, in the HA branch a shutdown ends the process in `release_service()` before `wrapped_master_service.reset()`, through `shutdown_exit` and `ExitAfterShutdown` (`_exit`, "no destructors run"): https://github.com/kvcache-ai/Mooncake/blob/8c19b7286c5ef46a5f2bf848f93acd86439a1b4e/mooncake-store/src/ha/leadership/master_service_supervisor.cpp#L733-L742 (`shutdown_exit` is at L437-L444, `ExitAfterShutdown` in `include/shutdown_request.h` L39-L42.) The non-HA signal handling came from #2359. That PR's goal was to stop the RPC server and the HTTP metadata server cleanly and to join the metrics thread. It did not discuss the index teardown. #### Observed Median of 3 runs; the range is in parentheses. Times are seconds after the signal. | | 12M objects | 25M objects | | -- | -- | -- | | Master RSS before the signal | 14.5 GiB | 30.2 GiB | | **SIGTERM to process exit** | **20.7** (20.1–20.9) | **48.2** (44.4–50.5) | | Inside it: RPC server stopped (log line `Shutdown signal received`) | 0.01–0.08 | 0.01–0.07 | | Inside it: `~MasterService` threads joined (log line `Client offboarding worker stopped`) | 0.84–0.88 | 0.11–0.19 | | Inside it: from there until `HTTP metadata server stopped`, which runs after the service is destroyed | 18.8 (18.3–19.0) | 46.0 (42.1–48.4) | | Inside it: from there until exit (static destructors and the kernel's teardown) | 1.05–1.07 | 1.94–2.13 | | Control: SIGKILL to process exit | 1.07 (0.99–1.11) | 2.37 (2.32–2.40) | | Master pinned to one NUMA node, SIGTERM to exit (1 run) | 13.0 | 25.9 | The exiting process uses one core the whole time. In the 12M run 1 it used 13.5 s of user time and 5.8 s of system time. In the 25M run 1 it used 28.3 s and 15.8 s. Pinned to one NUMA node (`numactl --cpunodebind=0 --membind=0`), system time dropped to 0.6 s (12M) and 1.2 s (25M). The user time stayed at about 1 µs per object. **Profile** (12M objects; `perf record --call-graph dwarf` at 499 Hz from 1 s before the signal to exit). The `mooncake_master` target is linked with `-s`, so for this run we relinked the same object files without `-s`; the `.text` section is identical. 89.5% of the process's on-CPU samples are under: ```text main -> ~shared_ptr<WrappedMasterService> -> MasterService::~MasterService() -> std::array<MasterService::MetadataShard, 1024>::~array() -> unordered_map<TenantId, TenantState> -> unordered_map<std::string, ObjectRecord>::~unordered_map() -> ~ObjectRecord -> ~ObjectMetadata -> ~vector<ObjectPart> -> ~vector<Replica> -> ~AllocatedBuffer / ~OffsetAllocationHandle (weak_ptr lock and release), free() ``` Self time: the hash-node deallocation loop 34.0%, `~AllocatedBuffer` 7.0%, `shared_ptr` control-block release 6.9%, libc (`free` and unresolved internals) 17.7%, `~OffsetAllocationHandle` 3.0%, the key-count gauge 0.6%. The kernel had 30% of all samples: 24% in page faults, of which 15% are automatic NUMA balancing faults (`do_numa_page`, with 12.5% in `migrate_misplaced_folio`) as the single exiting thread walks memory that the RPC threads allocated on both sockets, and 5% in the address-space teardown at exit (`exit_mmap`). For reference, we had measured 21.9 s for the same graceful exit on an older build based on upstream `245e7106`, with 12.17M objects (most with a local-disk replica, 7.0M of them disk-only), real stores and a 4-CPU limit. The setups differ, but the current `main` is not faster. #### Setup - `main` at `8c19b728`, `CMAKE_BUILD_TYPE=Release` (`-O3 -DNDEBUG`), `-DUSE_HTTP=ON`, no etcd, default glibc malloc. - Master flags: `--rpc_port=50051 --metrics_port=9003 --enable_http_metadata_server=true --http_metadata_server_host=0.0.0.0 --http_metadata_server_port=8080 --client_ttl=60 -enable_offload=true`. All other flags were defaults (standalone, offset allocator, 16 RPC threads). - Index: 12 fake segments × 675 GiB, mounted through `MountSegment` by a test client that keeps sending `Ping` until the end (metadata only, no data, no transfer engine). Objects are put with `BatchPutStart`/`BatchPutEnd`: 256 KiB each, 96-byte keys, one memory replica, no disk replicas. 12.0M and 25.0M objects use 36% and 75% of the capacity, so no eviction ran. We used the load tool attached to #4818 (`--mode=segments`, then `--mode=fill --threads=48 --put_batch=64 --value_size=262144`). Filling took 35–40 s (12M) and 80–100 s (25M). - Host: 2 × Intel Xeon Platinum 8462Y+ (128 logical CPUs, 2 NUMA nodes), Linux 6.8, `kernel.numa_balancing=1`. The master ran in a container that was not CPU-limited and was not pinned, except in the two pinned runs. - We measured from the signal to process exit with the shell's `wait` on the master's PID. The shell was the master's parent. Kubernetes was not in the loop, so kubelet and runtime overheads are not included. - 5 s idle between the end of the fill and the signal. #### Limits - One host type. The unpinned numbers depend on the host's NUMA layout and on NUMA balancing; the pinned case (about 1 µs per object) is closer to a single-socket host. We ran the pinned case once per size. - The 25M runs grew from run to run (44.4, 48.2, 50.5 s, and 55.3 s in a later run under `perf`). We did not find the cause. The 12M runs stayed within 0.8 s of each other. - Only memory replicas, one per object. Disk replicas and multi-replica objects add destructor work per replica that we did not measure on this build. - We did not try other allocators (jemalloc, tcmalloc) or glibc tunables. #### Possible directions (suggestions only) 1. In the non-HA branch, end the process without destroying the index once the RPC server and the service threads have stopped, as the HA branch does since #4745. The explicit steps from #2359 (stop the RPC server and the HTTP metadata server, join the threads) and a log flush could run first, then `_exit(0)`. With the measurements above, the exit would then take about as long as the thread joins plus the SIGKILL time (about 2–2.5 s in these runs). We have not checked whether any destructor further down has an effect outside the process that would need to run first. 2. A narrower variant: on the shutdown path only, let `MasterService` leave `metadata_shards_` undestroyed (for example, move the maps into a heap object that is intentionally not freed before process exit), and keep the rest of the destructors. 3. Destroying the 1,024 shards in parallel would shorten the wall time but keep the work. We list it only for completeness. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi commented

2026-10-29 21:24:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 22, 2026):

Thanks for opening this issue, @Juhyun-Kim-Memphis!

Field	Value
Issue	#4856
GitHub user ID	`32131411`
Reporter	@Juhyun-Kim-Memphis

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 22, 2026): Thanks for opening this issue, @Juhyun-Kim-Memphis! | Field | Value | |-------|-------| | **Issue** | #4856 | | **GitHub user ID** | `32131411` | | **Reporter** | @Juhyun-Kim-Memphis | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:16 +00:00

Author

Owner

@huangxiaobai-ydmy commented on GitHub (Oct 22, 2026):

Hi, I'd like to take this one, following direction 1 (end the non-HA process without destroying the index once the RPC server, the HTTP metadata server and the service threads have stopped, as the HA branch does since #4745).

I confirmed on current main that the non-HA branch returns 0 from main() after stopping the server, which destroys the full metadata_shards_ index on the way out. I'll audit the destructors that get skipped for any effect outside the process (non-HA has no etcd leases to revoke; the HTTP metadata server is stopped explicitly first) before choosing the exact exit point, and keep the explicit stop/join steps from #2359 in place so shutdown stays observable in the logs.

@huangxiaobai-ydmy commented on GitHub (Oct 22, 2026): Hi, I'd like to take this one, following direction 1 (end the non-HA process without destroying the index once the RPC server, the HTTP metadata server and the service threads have stopped, as the HA branch does since #4745). I confirmed on current main that the non-HA branch returns 0 from main() after stopping the server, which destroys the full metadata_shards_ index on the way out. I'll audit the destructors that get skipped for any effect outside the process (non-HA has no etcd leases to revoke; the HTTP metadata server is stopped explicitly first) before choosing the exact exit point, and keep the explicit stop/join steps from #2359 in place so shutdown stays observable in the logs.

bsakaguchi referenced this issue

2026-10-29 21:32:43 +00:00

[PR #1235] [CLOSED] Warning fix 1218 #2093