morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5898] [Bug] Nested output directories are not synced before durable publication #3578

New issue

Closed

opened 2026-10-31 00:42:03 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:03 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5898

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The shared exportWebScripts workflow returns success after publishing a script in a nested directory, but never syncs any of its nested directories. An instrumented filesystem test recorded successful real FileHandle.sync() calls for the script file, manifest.json, the export root and the root's parent only. The four intermediate directories received no sync.

The exported bytes were present and correct during the test. This reproduces the missing durability operations; it does not reproduce file loss after a crash.

Reproduction

Save the test below as tests/boundary/filesystem/outputTreeDurability.test.ts in the tested source checkout with locked dependencies installed, then run:

node node_modules/vitest/vitest.mjs run tests/boundary/filesystem/outputTreeDurability.test.ts --maxWorkers=1

This test was executed once on macOS. It invokes the same application workflow used by the CLI, using the repository's synthetic saved browser-scenario fixture. It requires no live browser. The wrapper delegates every actual filesystem operation and records each successful sync; it does not replace syncing with a no-op.

import { readFile, writeFile } from "node:fs/promises";
import { dirname, join, relative } from "node:path";
import { expect, it, onTestFinished, vi } from "vitest";

const calls = vi.hoisted(() => ({ synced: [] as string[] }));
vi.mock("node:fs/promises", async (importOriginal) => {
  const fs = await importOriginal<typeof import("node:fs/promises")>();
  return {
    ...fs,
    open: async (...args: Parameters<typeof fs.open>) => {
      const handle = await fs.open(...args);
      const sync = handle.sync.bind(handle);
      handle.sync = async () => {
        await sync();
        calls.synced.push(String(args[0]));
      };
      return handle;
    },
  };
});

import { exportWebScripts } from "../../../src/application/WebScriptExportService.js";
import { ArtifactResourceScope } from "../../../src/artifacts/ArtifactResourceScope.js";
import { webScriptExportResultSchema } from "../../../src/domain/webScriptExport.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { scriptScenarioFixture } from "../../fixtures/webScriptCapture.js";

it.skipIf(process.platform === "win32")("syncs all directories containing published script paths", async () => {
  const root = await createTestTempDirectory("rea-output-durability-");
  const resources = new ArtifactResourceScope();
  onTestFinished(() => resources.close());
  const input = {
    capture_path: join(root, "capture.json"),
    output_directory: join(root, "export"),
  };
  const bytes = Buffer.from("export const marker = 1;\n");
  await writeFile(input.capture_path, JSON.stringify(scriptScenarioFixture([
    { url: "https://fixture.test/app/main.js", bytes },
  ])));
  calls.synced.length = 0;
  const exported = await exportWebScripts(input, resources);
  if (!exported.ok) throw exported.error;
  const result = webScriptExportResultSchema.parse(exported.value.normalized_result);
  const script = result.scripts[0];
  if (script?.content.state !== "exported" || result.analysis_input === null)
    throw new Error("Expected exported script");
  const path = join(result.analysis_input.input_path, script.content.relative_path);
  expect(await readFile(path)).toEqual(bytes);
  const required: string[] = [];
  for (let directory = dirname(path); directory !== root; directory = dirname(directory))
    required.push(directory);
  console.log(JSON.stringify({
    ok: exported.ok,
    synced: calls.synced.map((entry) => relative(root, entry) || "."),
    missingDirectorySyncs: required.filter((entry) => !calls.synced.includes(entry)).map((entry) => relative(root, entry)),
  }, null, 2));
  expect(calls.synced).toEqual(expect.arrayContaining(required));
});

Result: the export succeeds and exact-byte readback succeeds, then the final assertion fails because the containing nested directories are missing from the sync calls.

Expected behavior

Before successful publication on hosts supporting directory fsync, sync the directories containing the newly written paths, including intermediate directories. The published contract says the output was durably committed, and commit says it syncs the owned output tree. The export guide also promises digest verification and durable readback.

Syncing file contents alone does not establish that the containing directory entries are durable. Syncing the output root and its parent does not explicitly sync entries inside deeper directories.

Impact and scope

P2: successful nested exports lack the stated directory-durability guarantee. A system crash before unsynced directory updates reach stable storage could leave a successful export's paths missing after recovery. This is a filesystem-dependent consequence inferred from the missing operations; no crash loss, incidence rate, or platform-specific failure probability was measured. Original input bytes were not modified.

The runtime observation covers script export on macOS. Source inspection shows the shared tree writer also serves artifact extraction. Linux crash behavior and other consumers were not executed. Windows intentionally skips directory fsync and is outside this report's scope. Outputs whose files are all direct children of the root do not have this particular nested-directory omission.

REA version

6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64, Node.js v24.22.0, locked Vitest v5.0.3. The exact repository Node pin v24.18.0 was not used.

Target and artifact context

One synthetic retained browser response for https://fixture.test/app/main.js, exported as a nested script plus a root-level manifest. No target program was executed.

Interface

CLI

Earliest failure stage

Cleanup, cancellation, or lifecycle

Relevant output or evidence

The focused test printed this observation; <origin> abbreviates the deterministic origin-directory digest:

{
  "ok": true,
  "synced": [
    "export/files/origins/<origin>/app/main.js",
    "export/manifest.json",
    "export",
    "."
  ],
  "missingDirectorySyncs": [
    "export/files/origins/<origin>/app",
    "export/files/origins/<origin>",
    "export/files/origins",
    "export/files"
  ]
}

One test failed at the final directory-sync assertion. This was a shared application/filesystem boundary run, rather than execution through CLI or MCP transport. Public reachability follows from the CLI command handler, which calls the tested workflow. The publisher writes scripts and manifest through the tree and commits before returning.

Typed outcome or recovery expected

Successful publication should wait for the required directory syncs. Failure to sync a required directory should remain a publication failure with the existing identity, cleanup and descriptor-ownership rules preserved.

Analysis or suspected cause

write() syncs each file before digest readback. #prepareParent records created nested directories, but its setup path does not sync them. commit() syncs only the root and parent, then marks the tree published. It validates nested-directory identity without syncing their entries.

Additional context

Fetched origin/main at c324d31b45fcab77e4ed7320ed60f0bf5a4ba188; the tree writer and script-export source paths cited above were unchanged from the tested revision. No crash injection or post-fix verification was performed.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5898 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The shared `exportWebScripts` workflow returns success after publishing a script in a nested directory, but never syncs any of its nested directories. An instrumented filesystem test recorded successful real `FileHandle.sync()` calls for the script file, `manifest.json`, the export root and the root's parent only. The four intermediate directories received no sync. The exported bytes were present and correct during the test. This reproduces the missing durability operations; it does not reproduce file loss after a crash. ### Reproduction Save the test below as `tests/boundary/filesystem/outputTreeDurability.test.ts` in the tested source checkout with locked dependencies installed, then run: ```sh node node_modules/vitest/vitest.mjs run tests/boundary/filesystem/outputTreeDurability.test.ts --maxWorkers=1 ``` This test was executed once on macOS. It invokes the same application workflow used by the CLI, using the repository's synthetic saved browser-scenario fixture. It requires no live browser. The wrapper delegates every actual filesystem operation and records each successful sync; it does not replace syncing with a no-op. ```ts import { readFile, writeFile } from "node:fs/promises"; import { dirname, join, relative } from "node:path"; import { expect, it, onTestFinished, vi } from "vitest"; const calls = vi.hoisted(() => ({ synced: [] as string[] })); vi.mock("node:fs/promises", async (importOriginal) => { const fs = await importOriginal<typeof import("node:fs/promises")>(); return { ...fs, open: async (...args: Parameters<typeof fs.open>) => { const handle = await fs.open(...args); const sync = handle.sync.bind(handle); handle.sync = async () => { await sync(); calls.synced.push(String(args[0])); }; return handle; }, }; }); import { exportWebScripts } from "../../../src/application/WebScriptExportService.js"; import { ArtifactResourceScope } from "../../../src/artifacts/ArtifactResourceScope.js"; import { webScriptExportResultSchema } from "../../../src/domain/webScriptExport.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { scriptScenarioFixture } from "../../fixtures/webScriptCapture.js"; it.skipIf(process.platform === "win32")("syncs all directories containing published script paths", async () => { const root = await createTestTempDirectory("rea-output-durability-"); const resources = new ArtifactResourceScope(); onTestFinished(() => resources.close()); const input = { capture_path: join(root, "capture.json"), output_directory: join(root, "export"), }; const bytes = Buffer.from("export const marker = 1;\n"); await writeFile(input.capture_path, JSON.stringify(scriptScenarioFixture([ { url: "https://fixture.test/app/main.js", bytes }, ]))); calls.synced.length = 0; const exported = await exportWebScripts(input, resources); if (!exported.ok) throw exported.error; const result = webScriptExportResultSchema.parse(exported.value.normalized_result); const script = result.scripts[0]; if (script?.content.state !== "exported" || result.analysis_input === null) throw new Error("Expected exported script"); const path = join(result.analysis_input.input_path, script.content.relative_path); expect(await readFile(path)).toEqual(bytes); const required: string[] = []; for (let directory = dirname(path); directory !== root; directory = dirname(directory)) required.push(directory); console.log(JSON.stringify({ ok: exported.ok, synced: calls.synced.map((entry) => relative(root, entry) || "."), missingDirectorySyncs: required.filter((entry) => !calls.synced.includes(entry)).map((entry) => relative(root, entry)), }, null, 2)); expect(calls.synced).toEqual(expect.arrayContaining(required)); }); ``` Result: the export succeeds and exact-byte readback succeeds, then the final assertion fails because the containing nested directories are missing from the sync calls. ### Expected behavior Before successful publication on hosts supporting directory fsync, sync the directories containing the newly written paths, including intermediate directories. The [`published` contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/SafeOutputTree.ts#L153-L156) says the output was durably committed, and [`commit`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/SafeOutputTree.ts#L272-L283) says it syncs the owned output tree. The [export guide](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/website-script-export.md#L30-L34) also promises digest verification and durable readback. Syncing file contents alone does not establish that the containing directory entries are durable. Syncing the output root and its parent does not explicitly sync entries inside deeper directories. ### Impact and scope P2: successful nested exports lack the stated directory-durability guarantee. A system crash before unsynced directory updates reach stable storage could leave a successful export's paths missing after recovery. This is a filesystem-dependent consequence inferred from the missing operations; no crash loss, incidence rate, or platform-specific failure probability was measured. Original input bytes were not modified. The runtime observation covers script export on macOS. Source inspection shows the shared tree writer also serves artifact extraction. Linux crash behavior and other consumers were not executed. Windows intentionally skips directory fsync and is outside this report's scope. Outputs whose files are all direct children of the root do not have this particular nested-directory omission. ### REA version 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64, Node.js v24.22.0, locked Vitest v5.0.3. The exact repository Node pin v24.18.0 was not used. ### Target and artifact context One synthetic retained browser response for `https://fixture.test/app/main.js`, exported as a nested script plus a root-level manifest. No target program was executed. ### Interface CLI ### Earliest failure stage Cleanup, cancellation, or lifecycle ### Relevant output or evidence The focused test printed this observation; `<origin>` abbreviates the deterministic origin-directory digest: ```json { "ok": true, "synced": [ "export/files/origins/<origin>/app/main.js", "export/manifest.json", "export", "." ], "missingDirectorySyncs": [ "export/files/origins/<origin>/app", "export/files/origins/<origin>", "export/files/origins", "export/files" ] } ``` One test failed at the final directory-sync assertion. This was a shared application/filesystem boundary run, rather than execution through CLI or MCP transport. Public reachability follows from the [CLI command handler](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/cli/webScriptCommands.ts#L18-L40), which calls the tested workflow. The [publisher](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/assets/PublishWebScripts.ts#L63-L117) writes scripts and manifest through the tree and commits before returning. ### Typed outcome or recovery expected Successful publication should wait for the required directory syncs. Failure to sync a required directory should remain a publication failure with the existing identity, cleanup and descriptor-ownership rules preserved. ### Analysis or suspected cause [`write()`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/SafeOutputTree.ts#L240-L257) syncs each file before digest readback. [`#prepareParent`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/SafeOutputTree.ts#L465-L513) records created nested directories, but its setup path does not sync them. [`commit()`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/SafeOutputTree.ts#L284-L313) syncs only the root and parent, then marks the tree published. It validates nested-directory identity without syncing their entries. ### Additional context Fetched `origin/main` at `c324d31b45fcab77e4ed7320ed60f0bf5a4ba188`; the tree writer and script-export source paths cited above were unchanged from the tested revision. No crash injection or post-fix verification was performed.

bsakaguchi closed this issue

2026-10-31 00:42:03 +00:00

bsakaguchi commented

2026-10-31 00:42:04 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

SafeOutputTree.commit now reverses the created-directory sequence, syncing children before parents, then root and root parent before publication. The boundary regression writes a/b/payload.txt using real filesystem syncs, checks the exact order/readback, and injects a parent sync failure to verify nonpublication/rollback. This establishes intended calls in test code, not crash-recovery durability measured on a real filesystem.

Implementation: SafeOutputTree.ts, safeOutputTreePublication.test.ts.

The relevant regression checks in safeOutputTreePublication.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. SafeOutputTree.commit now reverses the created-directory sequence, syncing children before parents, then root and root parent before publication. The boundary regression writes a/b/payload.txt using real filesystem syncs, checks the exact order/readback, and injects a parent sync failure to verify nonpublication/rollback. This establishes intended calls in test code, not crash-recovery durability measured on a real filesystem. Implementation: [SafeOutputTree.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/artifacts/SafeOutputTree.ts#L372), [safeOutputTreePublication.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/filesystem/safeOutputTreePublication.test.ts#L33). The relevant regression checks in [safeOutputTreePublication.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/filesystem/safeOutputTreePublication.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 00:48:18 +00:00

[GH-ISSUE #7077] Linux Hopper setup fails for current vendor release: fixed 120 s download deadline exceeded and feed main_exe_hash does not match the published launcher #4266

bsakaguchi referenced this issue

2026-10-31 00:48:18 +00:00

[GH-ISSUE #7077] Linux Hopper setup fails for current vendor release: fixed 120 s download deadline exceeded and feed main_exe_hash does not match the published launcher #4266

bsakaguchi referenced this issue

2026-10-31 00:52:25 +00:00

[GH-ISSUE #8138] [Bug] Hopper 6.6.1 Fedora RPM launcher still mismatches the feed main_exe_hash; deb launcher matches (REA 6.8.0) #4682

bsakaguchi referenced this issue

2026-10-31 01:09:51 +00:00

[PR #3590] [MERGED] fix(hopper)!: verify current Linux releases with the publisher key #5842