games-on-whales/LXC2Docker

Fork 0

mirror of https://github.com/games-on-whales/LXC2Docker.git synced 2026-10-15 12:52:04 +00:00

[PR #133] [MERGED] fix(oci): coalesce concurrent pulls of the same image ref #137

New issue

Closed

opened 2026-10-15 12:52:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:52:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/LXC2Docker/pull/133
Author: @JBailes
Created: 7/5/2026
Status: ✅ Merged
Merged: 7/5/2026
Merged by: @JBailes

Base: main ← Head: fix/coalesce-concurrent-pulls

📝 Commits (1)

b1df3b2 fix(oci): coalesce concurrent pulls of the same image ref

📊 Changes

3 files changed (+129 additions, -2 deletions)

View changed files

📝 go.mod (+2 -2)
📝 internal/lxc/manager.go (+30 -0)
➕ internal/lxc/pull_coalesce_test.go (+97 -0)

📄 Description

Problem

When two callers race to pull the same image ref, each independently runs skopeo+umoci into the same OCI store path. They corrupt each other's partial layers near completion, the wreckage is cleaned up, and the pull is retried — so with a caller that re-issues the pull (e.g. an orchestrator polling on a reconcile tick, or a docker pull overlapping a container-create auto-pull) it never converges.

Observed on a real deployment pulling a ~20 GB GPU image: 7+ concurrent skopeo copy processes for one ref, all writing the same path, colliding at ~16 GB every time, wiping the partial download, and restarting indefinitely — the image never landed and the box thrashed.

Fix

Wrap pullOCI in a singleflight.Group keyed by the image ref: the first caller runs the pull while any others block and share its result. Same ref → one pull; different refs → still parallel. This mirrors the existing per-image tmplLocks pattern used for template materialization, extending single-flighting to the pull (skopeo) itself, which is where the collision happened.

golang.org/x/sync was already an (indirect) dependency; it's now direct.

Tests

internal/lxc/pull_coalesce_test.go (run under -race):

concurrent same-ref pulls collapse to a single underlying run, all callers succeed;
distinct refs run in parallel (no false serialization).

A tiny pullImpl seam on Manager lets the wrapper be exercised without a real registry.

Validation

Built static and hot-deployed to the affected host: the previously-piling pull now runs with exactly one skopeo and completed the full ~20 GB download+unpack on the first try.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/LXC2Docker/pull/133 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 7/5/2026 **Status:** ✅ Merged **Merged:** 7/5/2026 **Merged by:** [@JBailes](https://github.com/JBailes) **Base:** `main` ← **Head:** `fix/coalesce-concurrent-pulls` --- ### 📝 Commits (1) - [`b1df3b2`](https://github.com/games-on-whales/LXC2Docker/commit/b1df3b20820cf2e62872ba0e2eebf2c32117ecba) fix(oci): coalesce concurrent pulls of the same image ref ### 📊 Changes **3 files changed** (+129 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `go.mod` (+2 -2) 📝 `internal/lxc/manager.go` (+30 -0) ➕ `internal/lxc/pull_coalesce_test.go` (+97 -0) </details> ### 📄 Description ## Problem When two callers race to pull the **same** image ref, each independently runs `skopeo`+`umoci` into the *same* OCI store path. They corrupt each other's partial layers near completion, the wreckage is cleaned up, and the pull is retried — so with a caller that re-issues the pull (e.g. an orchestrator polling on a reconcile tick, or a `docker pull` overlapping a container-create auto-pull) it never converges. Observed on a real deployment pulling a ~20 GB GPU image: **7+ concurrent `skopeo copy` processes** for one ref, all writing the same path, colliding at ~16 GB every time, wiping the partial download, and restarting indefinitely — the image never landed and the box thrashed. ## Fix Wrap `pullOCI` in a `singleflight.Group` keyed by the image ref: the first caller runs the pull while any others block and share its result. Same ref → one pull; different refs → still parallel. This mirrors the existing per-image `tmplLocks` pattern used for template materialization, extending single-flighting to the pull (skopeo) itself, which is where the collision happened. `golang.org/x/sync` was already an (indirect) dependency; it's now direct. ## Tests `internal/lxc/pull_coalesce_test.go` (run under `-race`): - concurrent same-ref pulls collapse to a **single** underlying run, all callers succeed; - distinct refs run in **parallel** (no false serialization). A tiny `pullImpl` seam on `Manager` lets the wrapper be exercised without a real registry. ## Validation Built static and hot-deployed to the affected host: the previously-piling pull now runs with exactly **one** skopeo and completed the full ~20 GB download+unpack on the first try. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:52:16 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

games-on-whales/LXC2Docker#137

No description provided.

Rows
Columns