[GH-ISSUE #448] Template soft-delete stalls permanently after marking `deleting=true` — re-issuing DELETE is a no-op, and deleting records still own aliases (blocks rebuilds, invisible to resolution) #110

New issue

Open

opened 2026-10-29 14:19:41 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:19:41 +00:00

Owner

Originally created by @mingfang on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/448

Summary

Follow-up to #447 (separate failure mode, same production deployment, 2026-10-27→28). Five templates (system, app-sandbox, fn-node, fn-extraction, fn-python) entered a state where:

Every rebuild is refused with 400 — alias 'X' already points to '<owner>', cannot rebind to '<new>' (the requestBuild pre-validation reads the alias row raw).
Resolution reports the alias missing: GET /templates/aliases/X → 404 template alias not found because resolve_alias filters out deleting records (catalog.rs:281), so clients see the name as free while rebuilds see it as taken.
No cleanup path exists: re-issuing DELETE /templates/{id} returns 204 but does nothing.

Root cause 1 — the stalled deletion has no resumer

The delete pipeline (catalog.rs:246-274) marks the record deleting=true, then deletes files → build log → record → releases the alias. In production the pipeline stalled immediately after the mark (the runtime process died mid-delete on 2026-10-27 ~18:49). Nothing retries it: no boot-time sweep, no reaper processes deleting=true records.

Root cause 2 — re-issuing DELETE is a silent no-op

catalog.rs:252 treats an already-deleting record as "another delete finished first" and returns Ok(()) early:

record => (None, record),          // already deleting → marked = None
...
let Some(record) = marked else {
    // Another delete finished first.
    return Ok(());                 // ← bails; steps 2-5 never run
};

The code comment at catalog.rs:264-265 ("Files go before the record, and the alias after it, so a failed delete can be retried by ID or alias") describes intent the implementation does not fulfill: a delete that failed after the mark cannot be retried — the retry sees deleting=true and returns success without cleaning anything. We verified this in production: five DELETE /templates/{id} calls returned 204 while the OSS documents were untouched (mtimes unchanged).

Observed end state (before manual repair)

For each of the five snapshots:

catalog/records/{id}.json present, deleting: true (36h stale)
catalog/aliases/{alias}.json present, pointing at the deleting id
resolution 404s, rebuilds refused, template list shows neither

We repaired by completing the pipeline manually against OSS (delete artifacts/{id}/, build log, record, then the alias row — the pipeline's own order).

Suggested fixes

Resume stalled deletions instead of bailing: when the record is already deleting=true, proceed with steps 2-5 (they are all idempotent — blob prefix delete, document deletes, release_alias). This single change makes the documented "retried by ID or alias" true.
A boot-time or periodic sweep over catalog/records/ for deleting=true records older than a threshold, completing the pipeline (protects against a crash between mark and completion forever).
Make the asymmetry loud: claim_alias's owner check should treat a deleting owner the way resolve_alias does (or at least the requestBuild 400 should say "alias owned by a snapshot that is being deleted — retry when deletion completes"), so clients don't deadlock between a refusal and a deletion that never finishes.

Environment: v0.2.4, single node + gateway, OSS (S3-compatible) repository, k8s deployment with the gateway and scheduler as separate services.

Originally created by @mingfang on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/448 ## Summary Follow-up to #447 (separate failure mode, same production deployment, 2026-10-27→28). Five templates (`system`, `app-sandbox`, `fn-node`, `fn-extraction`, `fn-python`) entered a state where: - **Every rebuild is refused** with `400 — alias 'X' already points to '<owner>', cannot rebind to '<new>'` (the `requestBuild` pre-validation reads the alias row raw). - **Resolution reports the alias missing**: `GET /templates/aliases/X` → `404 template alias not found` because `resolve_alias` filters out `deleting` records (`catalog.rs:281`), so clients see the name as *free* while rebuilds see it as *taken*. - **No cleanup path exists**: re-issuing `DELETE /templates/{id}` returns 204 but does nothing. ## Root cause 1 — the stalled deletion has no resumer The delete pipeline (`catalog.rs:246-274`) marks the record `deleting=true`, then deletes files → build log → record → releases the alias. In production the pipeline stalled immediately after the mark (the runtime process died mid-delete on 2026-10-27 ~18:49). Nothing retries it: no boot-time sweep, no reaper processes `deleting=true` records. ## Root cause 2 — re-issuing DELETE is a silent no-op `catalog.rs:252` treats an already-`deleting` record as "another delete finished first" and returns `Ok(())` early: ```rust record => (None, record), // already deleting → marked = None ... let Some(record) = marked else { // Another delete finished first. return Ok(()); // ← bails; steps 2-5 never run }; ``` The code comment at `catalog.rs:264-265` ("Files go before the record, and the alias after it, so a failed delete can be retried by ID or alias") describes intent the implementation does not fulfill: a delete that failed **after** the mark cannot be retried — the retry sees `deleting=true` and returns success without cleaning anything. We verified this in production: five `DELETE /templates/{id}` calls returned 204 while the OSS documents were untouched (mtimes unchanged). ## Observed end state (before manual repair) For each of the five snapshots: - `catalog/records/{id}.json` present, `deleting: true` (36h stale) - `catalog/aliases/{alias}.json` present, pointing at the deleting id - resolution 404s, rebuilds refused, template list shows neither We repaired by completing the pipeline manually against OSS (delete `artifacts/{id}/`, build log, record, then the alias row — the pipeline's own order). ## Suggested fixes 1. **Resume stalled deletions instead of bailing**: when the record is already `deleting=true`, proceed with steps 2-5 (they are all idempotent — blob prefix delete, document deletes, `release_alias`). This single change makes the documented "retried by ID or alias" true. 2. **A boot-time or periodic sweep** over `catalog/records/` for `deleting=true` records older than a threshold, completing the pipeline (protects against a crash between mark and completion forever). 3. **Make the asymmetry loud**: `claim_alias`'s owner check should treat a `deleting` owner the way `resolve_alias` does (or at least the `requestBuild` 400 should say "alias owned by a snapshot that is being deleted — retry when deletion completes"), so clients don't deadlock between a refusal and a deletion that never finishes. Environment: v0.2.4, single node + gateway, OSS (S3-compatible) repository, k8s deployment with the gateway and scheduler as separate services.

bsakaguchi commented

2026-10-29 14:19:42 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Oct 28, 2026):

Thanks for your report! This should have been fixed by commit d3818b085a.

@yingdi-shan commented on GitHub (Oct 28, 2026): Thanks for your report! This should have been fixed by commit d3818b085a20628c44e13d098613f9f9c45ab6b2.

bsakaguchi referenced this issue

2026-10-29 14:20:46 +00:00

[PR #110] [CLOSED] perf(network): skip empty egress programming on clean slots #186