morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5506] [Bug] WASM comparison budget refusals incorrectly tell callers to retry #3352

New issue

Closed

opened 2026-10-31 00:39:27 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:39:27 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5506

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

node scripts/rea.mjs compare-wasm-interfaces INPUT.json --format json correctly refuses an exports comparison above its fixed 32 MiB typed-interface representation allowance, but returns retryable: true and tells callers to wait for an active analysis. An independent verifier observed two byte-identical refusals for unchanged Evidence with 12,000 unique exports per side; a 10,000-export control succeeded. No concurrent inspection or active analysis provider was started.

Reproduction

From the baseline checkout with locked dependencies and the built CLI prepared, run the following exact script independently replayed by the verifier. It builds integrity-valid portable Evidence from the repository's source-owned fixture and exercises the supported CLI with isolated HOME and XDG directories. These are simulated producer facts, not an actual WABT inspection of a module. No external engine or live API is used.

python3 - <<'PYTHON'
import json, os, pathlib, subprocess, tempfile
work = pathlib.Path(tempfile.mkdtemp(prefix="rea-wasm-check-",
                                    dir=os.environ.get("REA_VERIFY_ROOT")))
env = {k: v for k, v in os.environ.items() if not k.startswith("REA_")}
for key, name in [("HOME", "home"), ("XDG_CONFIG_HOME", "config"),
                  ("XDG_DATA_HOME", "data"), ("XDG_CACHE_HOME", "cache"),
                  ("XDG_STATE_HOME", "state")]:
    folder = work / name
    folder.mkdir()
    env[key] = str(folder)
env["VERIFY_OUTPUT"] = str(work)
producer = r'''import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { createEvidence } from './src/domain/evidence.ts';
import { wasmArtifactFixture } from './tests/fixtures/wasm/artifact.ts';
for (const count of [10000, 12000]) {
  const report = wasmArtifactFixture(join(process.env.VERIFY_OUTPUT, 'module.wasm'));
  report.interface = { state: 'complete', issues: [], entries: Array.from({length:count}, (_, i) => ({
    direction:'export', module:null, name:`x${String(i).padStart(5,'0')}`, kind:'func', index:0,
    signature:{kind:'func',params:[],results:[]}, wat_form:`(export "x${String(i).padStart(5,'0')}" (func 0))`, issue:null
  })) };
  const evidence = createEvidence(
    {path:report.artifact.path, sha256:report.artifact.sha256, format:'file'},
    {id:'typed-fixture',name:'Typed fixture',version:'1'},
    {operation:'inspect_wasm_artifact',parameters:{path:report.artifact.path},result:report,confidence:'observed'});
  writeFileSync(join(process.env.VERIFY_OUTPUT, `${count}.json`),
    JSON.stringify({left:evidence,right:evidence,scope:'exports'}));
}
'''
created = subprocess.run(["node", "--import", "tsx", "--input-type=module", "-"],
                         input=producer, env=env, capture_output=True, text=True)
assert created.returncode == 0, created.stderr
for count, attempt in [(10000, 1), (12000, 1), (12000, 2)]:
    result = subprocess.run(["node", "scripts/rea.mjs", "compare-wasm-interfaces",
                             str(work / f"{count}.json"), "--format", "json"],
                            env=env, capture_output=True, text=True, timeout=90)
    (work / f"{count}-{attempt}.stdout.json").write_text(result.stdout)
    (work / f"{count}-{attempt}.stderr").write_text(result.stderr)
    value = json.loads(result.stdout)
    observed = {"entries_per_side": count, "attempt": attempt,
                "exit": result.returncode, "stderr": result.stderr}
    if result.returncode:
        observed.update(value)
    else:
        report = value["normalized_result"]
        observed.update(coverage=report["coverage"], summary=report["summary"])
    print(json.dumps(observed))
assert ((work / "12000-1.stdout.json").read_bytes() ==
        (work / "12000-2.stdout.json").read_bytes())
PYTHON

The control exited 0 with empty stderr, coverage: "complete-within-inputs", 10,000 unchanged entries and zero added, removed, changed or unknown entries. Both larger attempts exited 1 with empty stderr and identical stdout. The verifier's initial execution and exact-script replay produced the same distinction.

Expected behavior

Preserve the intentional fixed-budget refusal and numeric allowance, but report retryable: false and recovery advice identifying that fixed comparison representation budget. Waiting for another analysis cannot admit the same Evidence and scope.

This expectation is source-derived: src/domain/analysisErrorCore.ts:36-41 defines retryability as whether retrying the same request can succeed; src/domain/wasm/wasmInterface.ts:89-108 fixes the representation allowance at 33,554,432 bytes. docs/wasm-artifacts.md:115-134 supports portable complete Evidence with explicit comparison scope and states that comparison needs neither artifact-file reads nor an active toolchain. Unlimited or above-budget comparison is not promised.

Impact and scope

P2: incorrect machine-readable retryability can direct clients into repeat attempts that waiting cannot make successful. This is bounded to comparisons exceeding the fixed representation allowance. No automatic retry loop, data loss, service outage or incorrect successful comparison was observed.

Callers can recognize this specific message and maximum_bytes allowance and treat the refusal as permanent. The smaller control fits but does not preserve the larger input's coverage. MCP, actual WABT-produced inputs, installed-package behavior and other platforms were not independently replayed.

REA version

6.7.0; source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8, with the prepared built CLI runtime.

Runtime and platform

Node.js v26.12.0; Darwin arm64. No external analysis engine was used.

Target and artifact context

Portable inspect_wasm_artifact Evidence generated with createEvidence and tests/fixtures/wasm/artifact.ts, following the seam in tests/acceptance/wasmInterfaceComparison.test.ts:18-70. Both sides receive the same complete, authenticated fixture Evidence with explicit scope: "exports": 10,000 or 12,000 unique function exports, each with empty parameters/results. Recorded provider provenance is simulated.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Both 12,000-export CLI attempts produced the following fields; stdout was byte-identical and stderr was empty:

{
  "code": "resource_constraint",
  "category": "resource_constraint",
  "message": "WASM comparison exceeds typed interface representation capacity",
  "retryable": true,
  "remediation": {
    "action": "Wait for an active analysis to complete, then retry within the reported admission limits."
  },
  "details": {
    "operation": "compare_wasm_interfaces",
    "resource": "capacity",
    "reason": "WASM comparison exceeds typed interface representation capacity",
    "reported_limits": { "maximum_bytes": 33554432 }
  }
}

The passing control establishes that the fixture reaches the supported comparison path within its allowance. This allowance measures conservative retained/JSON representation accounting, not actual resident memory pressure or serialized request bytes.

Typed outcome or recovery expected

Keep resource_constraint, the 33,554,432-byte bound and intentional refusal. Return retryable: false and budget-specific recovery guidance rather than active-analysis admission advice. Genuine temporary capacity admission elsewhere should keep truthful retry semantics.

Analysis or suspected cause

Source-derived reasoning: src/cli/wasmCommands.ts:23-47 reaches the comparison and projects its error. src/application/wasm/WasmInterfaceComparisonService.ts:118-136 charges selected entries and per-group overhead against the fixed allowance, then constructs a capacity error without retryability or remediation overrides. src/domain/analysisErrorProjection.ts:109-115 defaults capacity to retryable; src/domain/analysisErrorPresentation.ts:67-78 supplies the observed wait-and-retry advice. A shared advice-text change alone would leave the retryable flag wrong; globally disabling capacity retries would affect temporary admission failures.

Additional context

Prepared with AI assistance and independently reproduced through the supported built CLI. Submission fetched upstream main at c374c2fe7290b914d73d9e27af38b498753ec593; no commits after the tested baseline touched the cited contract or failure-source files. Current-main applicability was source-checked, not separately runtime-reproduced there.

Related but distinct: #5504 concerns Rust compiler-hash capacity; #5405 concerns Rust/IL2CPP input admission; #5456 concerns SQLite worker replies; #5459 concerns HAR analysis-view accounting; #5460 concerns IL2CPP complete-report budgeting. Each has a different producer and caller trigger. PR #2694 changes WABT subprocess output capture, a different budget from this comparison representation allowance.

Regression range: NOT_ENOUGH_INFO; no known-good release or first-bad revision was established.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5506 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior `node scripts/rea.mjs compare-wasm-interfaces INPUT.json --format json` correctly refuses an exports comparison above its fixed 32 MiB typed-interface representation allowance, but returns `retryable: true` and tells callers to wait for an active analysis. An independent verifier observed two byte-identical refusals for unchanged Evidence with 12,000 unique exports per side; a 10,000-export control succeeded. No concurrent inspection or active analysis provider was started. ### Reproduction From the baseline checkout with locked dependencies and the built CLI prepared, run the following exact script independently replayed by the verifier. It builds integrity-valid portable Evidence from the repository's source-owned fixture and exercises the supported CLI with isolated HOME and XDG directories. These are simulated producer facts, not an actual WABT inspection of a module. No external engine or live API is used. ```sh python3 - <<'PYTHON' import json, os, pathlib, subprocess, tempfile work = pathlib.Path(tempfile.mkdtemp(prefix="rea-wasm-check-", dir=os.environ.get("REA_VERIFY_ROOT"))) env = {k: v for k, v in os.environ.items() if not k.startswith("REA_")} for key, name in [("HOME", "home"), ("XDG_CONFIG_HOME", "config"), ("XDG_DATA_HOME", "data"), ("XDG_CACHE_HOME", "cache"), ("XDG_STATE_HOME", "state")]: folder = work / name folder.mkdir() env[key] = str(folder) env["VERIFY_OUTPUT"] = str(work) producer = r'''import { writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { createEvidence } from './src/domain/evidence.ts'; import { wasmArtifactFixture } from './tests/fixtures/wasm/artifact.ts'; for (const count of [10000, 12000]) { const report = wasmArtifactFixture(join(process.env.VERIFY_OUTPUT, 'module.wasm')); report.interface = { state: 'complete', issues: [], entries: Array.from({length:count}, (_, i) => ({ direction:'export', module:null, name:`x${String(i).padStart(5,'0')}`, kind:'func', index:0, signature:{kind:'func',params:[],results:[]}, wat_form:`(export "x${String(i).padStart(5,'0')}" (func 0))`, issue:null })) }; const evidence = createEvidence( {path:report.artifact.path, sha256:report.artifact.sha256, format:'file'}, {id:'typed-fixture',name:'Typed fixture',version:'1'}, {operation:'inspect_wasm_artifact',parameters:{path:report.artifact.path},result:report,confidence:'observed'}); writeFileSync(join(process.env.VERIFY_OUTPUT, `${count}.json`), JSON.stringify({left:evidence,right:evidence,scope:'exports'})); } ''' created = subprocess.run(["node", "--import", "tsx", "--input-type=module", "-"], input=producer, env=env, capture_output=True, text=True) assert created.returncode == 0, created.stderr for count, attempt in [(10000, 1), (12000, 1), (12000, 2)]: result = subprocess.run(["node", "scripts/rea.mjs", "compare-wasm-interfaces", str(work / f"{count}.json"), "--format", "json"], env=env, capture_output=True, text=True, timeout=90) (work / f"{count}-{attempt}.stdout.json").write_text(result.stdout) (work / f"{count}-{attempt}.stderr").write_text(result.stderr) value = json.loads(result.stdout) observed = {"entries_per_side": count, "attempt": attempt, "exit": result.returncode, "stderr": result.stderr} if result.returncode: observed.update(value) else: report = value["normalized_result"] observed.update(coverage=report["coverage"], summary=report["summary"]) print(json.dumps(observed)) assert ((work / "12000-1.stdout.json").read_bytes() == (work / "12000-2.stdout.json").read_bytes()) PYTHON ``` The control exited 0 with empty stderr, `coverage: "complete-within-inputs"`, 10,000 unchanged entries and zero added, removed, changed or unknown entries. Both larger attempts exited 1 with empty stderr and identical stdout. The verifier's initial execution and exact-script replay produced the same distinction. ### Expected behavior Preserve the intentional fixed-budget refusal and numeric allowance, but report `retryable: false` and recovery advice identifying that fixed comparison representation budget. Waiting for another analysis cannot admit the same Evidence and scope. This expectation is source-derived: `src/domain/analysisErrorCore.ts:36-41` defines retryability as whether retrying the same request can succeed; `src/domain/wasm/wasmInterface.ts:89-108` fixes the representation allowance at 33,554,432 bytes. `docs/wasm-artifacts.md:115-134` supports portable complete Evidence with explicit comparison scope and states that comparison needs neither artifact-file reads nor an active toolchain. Unlimited or above-budget comparison is not promised. ### Impact and scope P2: incorrect machine-readable retryability can direct clients into repeat attempts that waiting cannot make successful. This is bounded to comparisons exceeding the fixed representation allowance. No automatic retry loop, data loss, service outage or incorrect successful comparison was observed. Callers can recognize this specific message and `maximum_bytes` allowance and treat the refusal as permanent. The smaller control fits but does not preserve the larger input's coverage. MCP, actual WABT-produced inputs, installed-package behavior and other platforms were not independently replayed. ### REA version 6.7.0; source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`, with the prepared built CLI runtime. ### Runtime and platform Node.js v26.12.0; Darwin arm64. No external analysis engine was used. ### Target and artifact context Portable `inspect_wasm_artifact` Evidence generated with `createEvidence` and `tests/fixtures/wasm/artifact.ts`, following the seam in `tests/acceptance/wasmInterfaceComparison.test.ts:18-70`. Both sides receive the same complete, authenticated fixture Evidence with explicit `scope: "exports"`: 10,000 or 12,000 unique function exports, each with empty parameters/results. Recorded provider provenance is simulated. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Both 12,000-export CLI attempts produced the following fields; stdout was byte-identical and stderr was empty: ```json { "code": "resource_constraint", "category": "resource_constraint", "message": "WASM comparison exceeds typed interface representation capacity", "retryable": true, "remediation": { "action": "Wait for an active analysis to complete, then retry within the reported admission limits." }, "details": { "operation": "compare_wasm_interfaces", "resource": "capacity", "reason": "WASM comparison exceeds typed interface representation capacity", "reported_limits": { "maximum_bytes": 33554432 } } } ``` The passing control establishes that the fixture reaches the supported comparison path within its allowance. This allowance measures conservative retained/JSON representation accounting, not actual resident memory pressure or serialized request bytes. ### Typed outcome or recovery expected Keep `resource_constraint`, the 33,554,432-byte bound and intentional refusal. Return `retryable: false` and budget-specific recovery guidance rather than active-analysis admission advice. Genuine temporary capacity admission elsewhere should keep truthful retry semantics. ### Analysis or suspected cause Source-derived reasoning: `src/cli/wasmCommands.ts:23-47` reaches the comparison and projects its error. `src/application/wasm/WasmInterfaceComparisonService.ts:118-136` charges selected entries and per-group overhead against the fixed allowance, then constructs a capacity error without retryability or remediation overrides. `src/domain/analysisErrorProjection.ts:109-115` defaults capacity to retryable; `src/domain/analysisErrorPresentation.ts:67-78` supplies the observed wait-and-retry advice. A shared advice-text change alone would leave the retryable flag wrong; globally disabling capacity retries would affect temporary admission failures. ### Additional context Prepared with AI assistance and independently reproduced through the supported built CLI. Submission fetched upstream main at `c374c2fe7290b914d73d9e27af38b498753ec593`; no commits after the tested baseline touched the cited contract or failure-source files. Current-main applicability was source-checked, not separately runtime-reproduced there. Related but distinct: #5504 concerns Rust compiler-hash capacity; #5405 concerns Rust/IL2CPP input admission; #5456 concerns SQLite worker replies; #5459 concerns HAR analysis-view accounting; #5460 concerns IL2CPP complete-report budgeting. Each has a different producer and caller trigger. PR #2694 changes WABT subprocess output capture, a different budget from this comparison representation allowance. Regression range: NOT_ENOUGH_INFO; no known-good release or first-bad revision was established.

bsakaguchi closed this issue

2026-10-31 00:39:28 +00:00

bsakaguchi commented

2026-10-31 00:39:28 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with sibling reports in Cluster AJ: #5504 ([Bug] Rust compiler-hash capacity refusals incorrectly tell callers to retry) and #5505. In both cases, permanent budget refusals erroneously advise callers that an unchanged retry will succeed.

Consolidated under Cluster AJ (Misleading Recovery Remediation on Capacity & Budget Refusals) in tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with sibling reports in **Cluster AJ**: #5504 (`[Bug] Rust compiler-hash capacity refusals incorrectly tell callers to retry`) and #5505. In both cases, permanent budget refusals erroneously advise callers that an unchanged retry will succeed. Consolidated under **Cluster AJ (Misleading Recovery Remediation on Capacity & Budget Refusals)** in tracker #4330.

bsakaguchi commented

2026-10-31 00:39:29 +00:00

Author

Owner

@morluto commented on GitHub (Oct 21, 2026):

Closing as fixed on main by github.com/morluto/rea@ef59174c1e, retained at 67a6477f01.

WASM comparison now marks its fixed representation byte budget non-retryable and names that allowance instead of advising the caller to wait. The aggregate-comparison regression exceeds the cap using individually admissible inventories and checks the projected guidance.

This audit inspected the current producer, regression source, issue discussion and main ancestry. Tests/runtime reproductions were not rerun, and no additional real-provider or platform coverage is implied.

@morluto commented on GitHub (Oct 21, 2026): Closing as fixed on main by https://github.com/morluto/rea/commit/ef59174c1edd6ed9adb0722b86a5b9a57217e310, retained at 67a6477f011ae358fc333e5a6e26c86d932654ed. WASM comparison now marks its fixed representation byte budget non-retryable and names that allowance instead of advising the caller to wait. The aggregate-comparison regression exceeds the cap using individually admissible inventories and checks the projected guidance. This audit inspected the current producer, regression source, issue discussion and main ancestry. Tests/runtime reproductions were not rerun, and no additional real-provider or platform coverage is implied.

bsakaguchi referenced this issue

2026-10-31 00:42:46 +00:00

[GH-ISSUE #5988] [Bug] Managed Evidence identity mismatches return invalid_request #3656

bsakaguchi referenced this issue

2026-10-31 01:10:40 +00:00

[PR #3853] [MERGED] fix(managed): project metadata with empty names without a display label #5949