[GH-ISSUE #432] Interrupted template deletes leave `deleting` tombstones that shadow aliases: start-by-name 400s, re-registration refused (v0.2.4) #104

New issue

Closed

opened 2026-10-29 14:19:37 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:37 +00:00

Owner

Originally created by @mingfang on GitHub (Oct 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/432

Summary

v0.2.4's routing refactor ("Route sandbox and build paths from one table") makes sandbox start-by-name resolve exclusively through the alias table (catalog/aliases/<name>.json → snapshot ID → record).

If a template delete is interrupted mid-flight, the registration record stays behind as a deleting: true tombstone. That tombstone:

Shadows the alias — every start-by-name fails with 400 template <name> not found, while the same payload with the registration UUID returns 201 (UUID start bypasses the alias table).
Blocks re-registration — claim_alias treats the tombstone as a live owner and rejects with an alias conflict, so affected templates cannot be re-pinned.
Cannot be deleted by name — name resolution skips deleting records, so DELETE /templates/<name> reports "not found".
Is invisible in GET /templates — the list filters out deleting records, so operators can't see what's wrong.
Defeats the stale-alias self-heal — alias_owner only releases a stale alias when the owner record is completely missing; a deleting record still exists, so the heal never fires.

In our production deployment (v0.2.4, OSS repository backend), 10 of 16 templates hit this over ~2 days of template churn. The failures tracked exactly the templates whose aliases pointed at tombstones; templates whose aliases pointed at live registrations (or were re-claimed fresh) worked normally throughout.

Code refs (v0.2.4)

src/snapshot/catalog.rs delete(): marks deleting=true, then deletes blobs → build log → record → alias. An interrupted run leaves the record marked. The comment says "A delete that failed partway leaves the record marked; retrying finishes it" — retrying by ID does finish it (verified: all our tombstones cleared with 204), but nothing retries automatically, and retry by name cannot even find the tombstone (resolution skips it).
resolve_alias() → alias_owner(): the self-heal (cleaning up stale alias pointing to missing snapshot) fires only when read_record() returns None. A deleting owner short-circuits the heal; the caller's !record.deleting filter then yields None → 404 on the aliases endpoint / 400 on start.
claim_alias(): read_record(existing).is_some() → alias_conflict, even when the existing record is a deleting tombstone.
SnapshotListFilter::templates(): excludes deleting records from GET /templates.

Reproduction

DELETE /templates/{id} for a template that owns an alias.
Interrupt the runtime mid-delete (e.g., restart the process between the record marking and the record/alias cleanup). We did not observe the interrupting event directly (candidates: process restart rolling over in-flight deletes, or long S3 blob cleanup outliving client timeouts), but three separate deletion batches left tombstones behind over two days.
Observe: GET /templates no longer lists the template; POST /sandboxes {"templateID": "<alias>"} → 400 template <alias> not found (single-digit ms); POST /v3/templates re-pin → alias conflict; GET /templates/aliases/<alias> → 404 template alias not found even though the alias document still exists in the store pointing at the tombstoned ID.

Recovery that worked

DELETE /templates/{tombstone_id} by UUID — the delete retry path resumes and finishes the partial delete.
Re-point or re-claim the alias. Since v0.2.4 has no names-scan fallback for resolution, an alias that was released (or manually re-pointed) stays unresolvable until a registration re-claims the name.

Suggested fix

resolve_alias / alias_owner: treat a deleting owner as absent — self-heal (release the alias, optionally finish the interrupted delete) instead of shadowing it.
claim_alias: allow stealing an alias whose owner record is deleting, same as the completely-missing case.
Optionally surface tombstones in the templates list (or sweep them at startup) so operators can detect and finish interrupted deletes.

Originally created by @mingfang on GitHub (Oct 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/432 ## Summary v0.2.4's routing refactor ("Route sandbox and build paths from one table") makes sandbox **start-by-name resolve exclusively through the alias table** (`catalog/aliases/<name>.json` → snapshot ID → record). If a template delete is interrupted mid-flight, the registration record stays behind as a `deleting: true` tombstone. That tombstone: 1. **Shadows the alias** — every start-by-name fails with `400 template <name> not found`, while the same payload with the registration UUID returns `201` (UUID start bypasses the alias table). 2. **Blocks re-registration** — `claim_alias` treats the tombstone as a live owner and rejects with an alias conflict, so affected templates cannot be re-pinned. 3. **Cannot be deleted by name** — name resolution skips `deleting` records, so `DELETE /templates/<name>` reports "not found". 4. **Is invisible in `GET /templates`** — the list filters out `deleting` records, so operators can't see what's wrong. 5. **Defeats the stale-alias self-heal** — `alias_owner` only releases a stale alias when the owner record is *completely missing*; a `deleting` record still exists, so the heal never fires. In our production deployment (v0.2.4, OSS repository backend), 10 of 16 templates hit this over ~2 days of template churn. The failures tracked exactly the templates whose aliases pointed at tombstones; templates whose aliases pointed at live registrations (or were re-claimed fresh) worked normally throughout. ## Code refs (v0.2.4) - `src/snapshot/catalog.rs` `delete()`: marks `deleting=true`, then deletes blobs → build log → record → alias. An interrupted run leaves the record marked. The comment says "A delete that failed partway leaves the record marked; retrying finishes it" — retrying **by ID** does finish it (verified: all our tombstones cleared with `204`), but nothing retries automatically, and retry **by name** cannot even find the tombstone (resolution skips it). - `resolve_alias()` → `alias_owner()`: the self-heal (`cleaning up stale alias pointing to missing snapshot`) fires only when `read_record()` returns `None`. A `deleting` owner short-circuits the heal; the caller's `!record.deleting` filter then yields `None` → `404` on the aliases endpoint / `400` on start. - `claim_alias()`: `read_record(existing).is_some()` → `alias_conflict`, even when the existing record is a `deleting` tombstone. - `SnapshotListFilter::templates()`: excludes `deleting` records from `GET /templates`. ## Reproduction 1. `DELETE /templates/{id}` for a template that owns an alias. 2. Interrupt the runtime mid-delete (e.g., restart the process between the record marking and the record/alias cleanup). We did not observe the interrupting event directly (candidates: process restart rolling over in-flight deletes, or long S3 blob cleanup outliving client timeouts), but three separate deletion batches left tombstones behind over two days. 3. Observe: `GET /templates` no longer lists the template; `POST /sandboxes {"templateID": "<alias>"}` → `400 template <alias> not found` (single-digit ms); `POST /v3/templates` re-pin → alias conflict; `GET /templates/aliases/<alias>` → `404 template alias not found` even though the alias document still exists in the store pointing at the tombstoned ID. ## Recovery that worked 1. `DELETE /templates/{tombstone_id}` by UUID — the delete retry path resumes and finishes the partial delete. 2. Re-point or re-claim the alias. Since v0.2.4 has no names-scan fallback for resolution, an alias that was released (or manually re-pointed) stays unresolvable until a registration re-claims the name. ## Suggested fix - `resolve_alias` / `alias_owner`: treat a `deleting` owner as absent — self-heal (release the alias, optionally finish the interrupted delete) instead of shadowing it. - `claim_alias`: allow stealing an alias whose owner record is `deleting`, same as the completely-missing case. - Optionally surface tombstones in the templates list (or sweep them at startup) so operators can detect and finish interrupted deletes.

bsakaguchi closed this issue

2026-10-29 14:19:38 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:43 +00:00

[PR #104] [MERGED] fix(snapshot): return error instead of panicking in write_commit_marker #179