morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6234] [Bug] package-e2e repository skill stage times out fetching skills@1.7.2 from the registry #3795

New issue

Closed

opened 2026-10-31 00:44:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:07 +00:00

Owner

Originally created by @kaoru0822-kitauji on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6234

Originally assigned to: @kaoru0822-kitauji on GitHub.

Observed behavior

The required package-e2e lane ("Native package baseline (linux x64)") intermittently fails at the repository skill stage of npm run verify:package:

[verify:package] repository skill: start            02:21:30
[verify:package] workspace cleanup: start           02:23:32
Error: Command failed: node npm-cli.js exec --yes --package skills@1.7.2 -- skills add /tmp/rea-package-*/repository --skill reverse-engineer-anything --agent codex --yes
  killed: true, signal: 'SIGTERM', stdout: '', stderr: ''

Seen on main CI run 39709973007 (ce5bc8cb), and on PR runs 39710915987 and 39711962559 within ~30 minutes.

The same stage's duration in passing runs from the same window: 4 702 ms, 27 101 ms and 117 695 ms (the stage has a 120 000 ms timeout).

Analysis

scripts/verify-repository-skill.mjs runs npm exec --yes --package skills@1.7.2, which resolves skills and its unlocked transitive dependencies (tar, yaml, …) against the npm registry on every run. Locally the skills add itself takes ~2 s against a full repository snapshot with a warm cache and ~7 s with a cold cache, so the variable 5–120 s cost is registry resolution, not skill discovery.

Expected behavior

The verifier should run the pinned Skills CLI from the lockfile install, with no registry access at verification time, so the stage is deterministic and its transitive dependencies are locked.

Originally created by @kaoru0822-kitauji on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6234 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Observed behavior The required `package-e2e` lane ("Native package baseline (linux x64)") intermittently fails at the `repository skill` stage of `npm run verify:package`: ``` [verify:package] repository skill: start 02:21:30 [verify:package] workspace cleanup: start 02:23:32 Error: Command failed: node npm-cli.js exec --yes --package skills@1.7.2 -- skills add /tmp/rea-package-*/repository --skill reverse-engineer-anything --agent codex --yes killed: true, signal: 'SIGTERM', stdout: '', stderr: '' ``` Seen on main CI run 39709973007 (`ce5bc8cb`), and on PR runs 39710915987 and 39711962559 within ~30 minutes. The same stage's duration in passing runs from the same window: 4 702 ms, 27 101 ms and 117 695 ms (the stage has a 120 000 ms timeout). ### Analysis `scripts/verify-repository-skill.mjs` runs `npm exec --yes --package skills@1.7.2`, which resolves `skills` and its unlocked transitive dependencies (`tar`, `yaml`, …) against the npm registry on every run. Locally the `skills add` itself takes ~2 s against a full repository snapshot with a warm cache and ~7 s with a cold cache, so the variable 5–120 s cost is registry resolution, not skill discovery. ### Expected behavior The verifier should run the pinned Skills CLI from the lockfile install, with no registry access at verification time, so the stage is deterministic and its transitive dependencies are locked.

bsakaguchi

2026-10-31 00:44:07 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-31 00:44:08 +00:00

Author

Owner

@aron-intframe commented on GitHub (Oct 22, 2026):

Your analysis is right that the cost is registry resolution, and there is a specific reason it can never be anything else on this one lane: linux x64 is the only platform whose npm download cache is never written by a job that runs this command, so the restored cache cannot contain the skills closure and every run is a full cold install.

.github/actions/npm-cache saves only when can_save and inputs.save are both true. package-e2e passes save: ${{ matrix.os != 'ubuntu-latest' }} (ci.yml:682), so the linux x64 lane takes the restore-only branch. The Linux x64 key npm-download-Linux-x64-<lockfiles hash> is written by static (ci.yml:131, save: "true", runs-on: ubuntu-latest), which runs npm ci and the lint/format checks and never calls npm exec --package skills@1.7.2. So npm_config_cache at ci.yml:706 points at a cache that is warm for the lockfile and cold for Skills. On windows/macos the matrix expression is true, so those lanes do write the Skills closure into their own platform key and self-warm on default-branch pushes. The slow runs should be concentrated on linux x64.

Priming from the lockfile cannot close the gap, because the specs float. skills@1.7.2 declares tar ^7.5.20 and yaml ^2.8.3. Your package.json pins yaml at 2.9.0; the range resolves to 2.9.1. With a cache holding exactly yaml@2.9.0 and tar@7.4.3, the run still performed 16 registry operations and still downloaded yaml-2.9.1.tgz and tar-7.5.22.tgz:

restored CI cache, as the lane runs today             567 ms  registry ops=16 (audit POST: 1)
      GET 200 https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz 92ms (cache miss)
      GET 200 https://registry.npmjs.org/tar/-/tar-7.5.22.tgz 92ms (cache miss)
      POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 190ms

Cold is 18 operations: 9 packument reads, 8 tarballs, and one audit POST. With the cache actually warm it is 1, and the stage drops from "whatever the registry feels like" to a quarter second:

A) fresh HOME, fresh npm cache (today)       602 / 601 / 570 ms   registry fetches: 18 / 18 / 18
B) cache kept across runs                    694 / 266 / 243 ms   registry fetches: 18 /  1 /  1
C) fresh HOME + shared npm_config_cache    87563 / 245 / 244 ms   registry fetches: 18 /  1 /  1

That 87.6 s is the same command on the same host; ten other cold runs on this box ran 551-735 ms (median 621 ms, registry ~30 ms away). A cold path with 18 serial registry operations has no floor, so 4 702 / 27 101 / 117 695 ms and a SIGTERM at 120 000 ms are all the same run with different luck. A hosted runner is further from the registry than this box, so your spread is wider than mine, not different in kind.

Two other things the measurements show:

--prefer-offline removes the last revalidation once the cache is warm, and the audit POST is pure cost here:

fresh HOME, warm cache                        241 ms  fetches=1   (GET /skills, cache revalidated)
fresh HOME, warm cache, --prefer-offline      213 ms  fetches=0
fresh HOME, warm cache, --offline             224 ms  fetches=0
empty cache, as today                         835 ms  registry ops=18  audit POST=yes
empty cache, + --no-audit --no-fund           557 ms  registry ops=17  audit POST=no

And the failure carries no diagnostics, which is why the three runs look identical. exec is promisify(execFile) with timeout: 120_000; npm writes nothing to a pipe at the default loglevel, so the kill produces exactly your error object. Reproduced with a 250 ms clock instead of 120 s:

--- default loglevel (what CI runs) ---
  killed  : true   signal: "SIGTERM"   code: null
  stdout  : ""
  stderr  : ""
--- --loglevel=http ---
  stderr  : "npm http fetch GET 200 .../skills 38ms (cache miss)\nnpm http fetch GET 200 .../skills 21ms (cache miss)\n"

Fixes, in the order I would apply them:

Your expected behaviour is the right one. Add skills@1.7.2 to devDependencies so tar/yaml land in package-lock.json, then call node ../../node_modules/skills/bin/cli.mjs add ... (the package declares bin: { skills, add-skill } -> bin/cli.mjs) with cwd still the consumer dir and the same HOME override. Zero registry access at verification time, and the transitive tree stops drifting between runs.
If you want the one-line mitigation first: drop save: ${{ matrix.os != 'ubuntu-latest' }} from package-e2e so the linux x64 lane writes its own key on default-branch pushes, and add --prefer-offline --no-audit --no-fund to the npm exec argv. Cache-hit runs then cost ~0.25 s; cache-miss runs still pay full price, so this is a mitigation, not the fix.
Independent of either: pass --loglevel=http (or stream the child's stderr) so a future timeout on a required lane leaves a trail instead of stdout: '', stderr: ''.

I checked (1) against the current call rather than assuming it. With a minimal snapshot holding one authored skill, the npx form and the lockfile-installed CLI produce byte-identical installs from the same argv and the same HOME override, and neither writes into the snapshot:

npx           228 ms  installed under consumer/.agents/skills: ["demo-skill/SKILL.md"]
lockfile       73 ms  installed under consumer/.agents/skills: ["demo-skill/SKILL.md"]
identical bytes: true
snapshot untouched by either: [".agents/skills/demo-skill/SKILL.md"]

Since you pass the snapshot path explicitly and --agent codex installs relative to cwd, moving the CLI inside the repo tree does not change what root discovery picks. The 73 ms is the whole stage with no registry contact at all.

We get bitten by the diagnostics point a lot: a scheduled job killed on timeout reports a summary containing none of the cause, so a required check fails without saying anything. Our write-up of that pattern is item 8 in our own catalog (ours): https://github.com/aron-intframe/agent-guards#8-read-the-raw-session-not-the-first-line-of-the-log

@aron-intframe commented on GitHub (Oct 22, 2026): Your analysis is right that the cost is registry resolution, and there is a specific reason it can never be anything else on this one lane: **linux x64 is the only platform whose npm download cache is never written by a job that runs this command**, so the restored cache cannot contain the `skills` closure and every run is a full cold install. `.github/actions/npm-cache` saves only when `can_save` *and* `inputs.save` are both true. `package-e2e` passes `save: ${{ matrix.os != 'ubuntu-latest' }}` (ci.yml:682), so the linux x64 lane takes the restore-only branch. The Linux x64 key `npm-download-Linux-x64-<lockfiles hash>` is written by `static` (ci.yml:131, `save: "true"`, `runs-on: ubuntu-latest`), which runs `npm ci` and the lint/format checks and never calls `npm exec --package skills@1.7.2`. So `npm_config_cache` at ci.yml:706 points at a cache that is warm for the lockfile and cold for Skills. On windows/macos the matrix expression is `true`, so those lanes do write the Skills closure into their own platform key and self-warm on default-branch pushes. The slow runs should be concentrated on linux x64. Priming from the lockfile cannot close the gap, because the specs float. `skills@1.7.2` declares `tar ^7.5.20` and `yaml ^2.8.3`. Your `package.json` pins `yaml` at `2.9.0`; the range resolves to `2.9.1`. With a cache holding exactly `yaml@2.9.0` and `tar@7.4.3`, the run still performed 16 registry operations and still downloaded `yaml-2.9.1.tgz` and `tar-7.5.22.tgz`: ``` restored CI cache, as the lane runs today 567 ms registry ops=16 (audit POST: 1) GET 200 https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz 92ms (cache miss) GET 200 https://registry.npmjs.org/tar/-/tar-7.5.22.tgz 92ms (cache miss) POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 190ms ``` Cold is 18 operations: 9 packument reads, 8 tarballs, and one audit POST. With the cache actually warm it is 1, and the stage drops from "whatever the registry feels like" to a quarter second: ``` A) fresh HOME, fresh npm cache (today) 602 / 601 / 570 ms registry fetches: 18 / 18 / 18 B) cache kept across runs 694 / 266 / 243 ms registry fetches: 18 / 1 / 1 C) fresh HOME + shared npm_config_cache 87563 / 245 / 244 ms registry fetches: 18 / 1 / 1 ``` That 87.6 s is the same command on the same host; ten other cold runs on this box ran 551-735 ms (median 621 ms, registry ~30 ms away). A cold path with 18 serial registry operations has no floor, so 4 702 / 27 101 / 117 695 ms and a SIGTERM at 120 000 ms are all the same run with different luck. A hosted runner is further from the registry than this box, so your spread is wider than mine, not different in kind. Two other things the measurements show: `--prefer-offline` removes the last revalidation once the cache is warm, and the audit POST is pure cost here: ``` fresh HOME, warm cache 241 ms fetches=1 (GET /skills, cache revalidated) fresh HOME, warm cache, --prefer-offline 213 ms fetches=0 fresh HOME, warm cache, --offline 224 ms fetches=0 empty cache, as today 835 ms registry ops=18 audit POST=yes empty cache, + --no-audit --no-fund 557 ms registry ops=17 audit POST=no ``` And the failure carries no diagnostics, which is why the three runs look identical. `exec` is `promisify(execFile)` with `timeout: 120_000`; npm writes nothing to a pipe at the default loglevel, so the kill produces exactly your error object. Reproduced with a 250 ms clock instead of 120 s: ``` --- default loglevel (what CI runs) --- killed : true signal: "SIGTERM" code: null stdout : "" stderr : "" --- --loglevel=http --- stderr : "npm http fetch GET 200 .../skills 38ms (cache miss)\nnpm http fetch GET 200 .../skills 21ms (cache miss)\n" ``` Fixes, in the order I would apply them: 1. Your expected behaviour is the right one. Add `skills@1.7.2` to `devDependencies` so `tar`/`yaml` land in `package-lock.json`, then call `node ../../node_modules/skills/bin/cli.mjs add ...` (the package declares `bin: { skills, add-skill }` -> `bin/cli.mjs`) with `cwd` still the consumer dir and the same `HOME` override. Zero registry access at verification time, and the transitive tree stops drifting between runs. 2. If you want the one-line mitigation first: drop `save: ${{ matrix.os != 'ubuntu-latest' }}` from `package-e2e` so the linux x64 lane writes its own key on default-branch pushes, and add `--prefer-offline --no-audit --no-fund` to the `npm exec` argv. Cache-hit runs then cost ~0.25 s; cache-miss runs still pay full price, so this is a mitigation, not the fix. 3. Independent of either: pass `--loglevel=http` (or stream the child's stderr) so a future timeout on a required lane leaves a trail instead of `stdout: '', stderr: ''`. I checked (1) against the current call rather than assuming it. With a minimal snapshot holding one authored skill, the npx form and the lockfile-installed CLI produce byte-identical installs from the same argv and the same `HOME` override, and neither writes into the snapshot: ``` npx 228 ms installed under consumer/.agents/skills: ["demo-skill/SKILL.md"] lockfile 73 ms installed under consumer/.agents/skills: ["demo-skill/SKILL.md"] identical bytes: true snapshot untouched by either: [".agents/skills/demo-skill/SKILL.md"] ``` Since you pass the snapshot path explicitly and `--agent codex` installs relative to `cwd`, moving the CLI inside the repo tree does not change what root discovery picks. The 73 ms is the whole stage with no registry contact at all. We get bitten by the diagnostics point a lot: a scheduled job killed on timeout reports a summary containing none of the cause, so a required check fails without saying anything. Our write-up of that pattern is item 8 in our own catalog (ours): https://github.com/aron-intframe/agent-guards#8-read-the-raw-session-not-the-first-line-of-the-log

bsakaguchi referenced this issue

2026-10-31 00:50:58 +00:00

[GH-ISSUE #7656] [Bug] Indonesian and Italian analysis tables omit the Linux x64 host prerequisite #4579

bsakaguchi referenced this issue

2026-11-01 21:10:53 +00:00

[GH-ISSUE #3582] [Tracker] Ready to close or merge: issues and pull requests (October 2026) #1330

bsakaguchi referenced this issue

2026-11-01 21:10:54 +00:00

[GH-ISSUE #3795] [Tracker] Open documentation issues and pull requests (October 2026) #1468

bsakaguchi referenced this issue

2026-11-01 21:10:58 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471