morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4797] [Bug] analyze-javascript-application keeps every semantic unknown past the node ceilings #2494

New issue

Closed

opened 2026-10-17 21:05:43 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-17 21:05:43 +00:00

Owner

Originally created by @JayOfTheKeyboard on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4797

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

analyze-javascript-application caps semantic graph nodes at the 20,000-per-file ceiling, but keeps every semantic unknown. Unknowns grow with the source after the node ceiling has cut in, so they become most of the result.

On a generated 1.66 MB file, the graph keeps 20,000 nodes and 160,000 unknowns. The unknowns are 99.9 MB of a 126.5 MB JSON result. Coverage reports truncated: true and lists only the two node ceilings under limits.

statements	source bytes	nodes	unknowns	unknowns JSON bytes	result bytes	wall s	peak RSS
1,000	39,690	7,001	4,000	2,625,835	9,074,770	5.6	410 MiB
10,000	406,890	20,000	40,000	24,908,043	41,914,821	18.5	594 MiB
40,000	1,660,890	20,000	160,000	99,932,043	126,543,218	45.0	903 MiB

Every statement gives four unknowns (two ambiguous-target and one dynamic-property in object-flow, one dynamic-call in call-flow), whether or not its nodes were kept. Each unknown ID then appears a second time under coverage.families[].unknown_ids.

Real Electron apps show the same shape. Discord's Linux app.asar (3.5 MB, 5 files, 3,074,650 text bytes read) gives 94,013 unknowns, about 59.7 MB of a 108 MB normalized_result, beside 20,968 nodes. Teams for Linux (17.7 MB app.asar, run with --integrity-policy record-and-continue) and Antigravity (2.1 MB app.asar beside 19 MB unpacked) gave 289 MB and 303 MB of TOON output, with 3.5 GiB and 3.0 GiB peak RSS.

Reproduction

case_root="$(mktemp -d)"
mkdir "$case_root/fixture"
python3 -c '
for i in range(40000):
    print(f"globalThis.u{i%50}.p{i}(globalThis.v{i%7}[k{i%3}]);")
' > "$case_root/fixture/main.js"
rea analyze-javascript-application "$case_root/fixture" --format json > "$case_root/result.json"
jq -r '.normalized_result.semantic_graph
  | "nodes=\(.nodes|length) unknowns=\(.unknowns|length) unknown_bytes=\(.unknowns|tojson|length) truncated=\(.coverage.truncated)"' \
  "$case_root/result.json"
# nodes=20000 unknowns=160000 unknown_bytes=99932043 truncated=true

Change range(40000) to range(1000) or range(10000) for the other rows.

Expected behavior

Unknowns have a budget, as nodes do. Past it, the graph records how many unknowns of each family and reason were left out, and lists the budget under coverage.limits. Then the result size follows the configured ceilings and not the source length.

Impact and scope

Any large or minified bundle. CLI users get a result tens of times larger than the input (76x for the 1.66 MB file). Agents reading the CLI output, and anything that keeps the complete Evidence, pay for the unknowns in time, memory and bytes. --max-semantic-nodes and --max-semantic-nodes-per-file do not reduce them: with both at 1,000, the 10,000-statement file keeps 1,000 nodes and all 40,000 unknowns.

REA version

rea-agents@6.5.0 (npm), and main at 2cf9c28f

Runtime and platform

Node 24.18.0, Fedora 44 x86_64

Target and artifact context

Generated JavaScript (recipe above). Real-app figures come from the Flathub packages of Discord and Teams for Linux, and from Antigravity's Linux install.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Unknown reasons on the 40,000-statement file:

80000  ambiguous-target  object-flow
40000  dynamic-call      call-flow
40000  dynamic-property  object-flow

The same file on 6.5.0: 126,545,939 bytes, 20,000 nodes, 160,000 unknowns, 69.5 s, 1.24 GiB peak RSS.

I also ran it on open PR #2092 (head b6a097b3), which replaces the default node ceilings. That run kept 280,001 nodes and the same 160,000 unknowns: 353,160,381 bytes, 160 s, 2.7 GiB peak RSS. On Discord, #2092 kept 317 unknowns, but 190,560 nodes and 249 MB. So #2092 does not bound this case.

No reference in unknowns dangles: every evidence.context_id, node_id and candidate_node_ids entry resolves, at 1,000 and at 40,000 statements.

Typed outcome or recovery expected

A completed result with coverage.truncated: true, a named unknowns limit in coverage.limits, and per-family omitted counts for the dropped unknowns.

Analysis or suspected cause

adoptFileProjection in src/application/javascript/JavaScriptSemanticGraphBuilder.ts (lines 334 to 355) checks new nodes against min(fileNodeCeiling, nodeCeiling - state.nodes.size) and throws if they exceed it. The loop just below adopts every projection.unknowns entry with no budget. recordUnavailableFile (lines 388 to 405) and the getter-expansion path (lines 570 to 592) add unknowns without one too.

I have not looked at where the projection decides to emit an unknown for a statement whose nodes were dropped.

Additional context

#1065 says complete Evidence keeps its unknowns, so if keeping all of them is intended, a page or a count-only view for unknowns would work instead. I can send a PR for whichever shape you prefer.

Originally created by @JayOfTheKeyboard on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4797 ### Before submitting - [X] I searched the open and closed issues for a related report. - [X] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `analyze-javascript-application` caps semantic graph nodes at the 20,000-per-file ceiling, but keeps every semantic unknown. Unknowns grow with the source after the node ceiling has cut in, so they become most of the result. On a generated 1.66 MB file, the graph keeps 20,000 nodes and 160,000 unknowns. The unknowns are 99.9 MB of a 126.5 MB JSON result. Coverage reports `truncated: true` and lists only the two node ceilings under `limits`. | statements | source bytes | nodes | unknowns | unknowns JSON bytes | result bytes | wall s | peak RSS | | --- | --- | --- | --- | --- | --- | --- | --- | | 1,000 | 39,690 | 7,001 | 4,000 | 2,625,835 | 9,074,770 | 5.6 | 410 MiB | | 10,000 | 406,890 | 20,000 | 40,000 | 24,908,043 | 41,914,821 | 18.5 | 594 MiB | | 40,000 | 1,660,890 | 20,000 | 160,000 | 99,932,043 | 126,543,218 | 45.0 | 903 MiB | Every statement gives four unknowns (two `ambiguous-target` and one `dynamic-property` in object-flow, one `dynamic-call` in call-flow), whether or not its nodes were kept. Each unknown ID then appears a second time under `coverage.families[].unknown_ids`. Real Electron apps show the same shape. Discord's Linux `app.asar` (3.5 MB, 5 files, 3,074,650 text bytes read) gives 94,013 unknowns, about 59.7 MB of a 108 MB `normalized_result`, beside 20,968 nodes. Teams for Linux (17.7 MB `app.asar`, run with `--integrity-policy record-and-continue`) and Antigravity (2.1 MB `app.asar` beside 19 MB unpacked) gave 289 MB and 303 MB of TOON output, with 3.5 GiB and 3.0 GiB peak RSS. ### Reproduction ```sh case_root="$(mktemp -d)" mkdir "$case_root/fixture" python3 -c ' for i in range(40000): print(f"globalThis.u{i%50}.p{i}(globalThis.v{i%7}[k{i%3}]);") ' > "$case_root/fixture/main.js" rea analyze-javascript-application "$case_root/fixture" --format json > "$case_root/result.json" jq -r '.normalized_result.semantic_graph | "nodes=\(.nodes|length) unknowns=\(.unknowns|length) unknown_bytes=\(.unknowns|tojson|length) truncated=\(.coverage.truncated)"' \ "$case_root/result.json" # nodes=20000 unknowns=160000 unknown_bytes=99932043 truncated=true ``` Change `range(40000)` to `range(1000)` or `range(10000)` for the other rows. ### Expected behavior Unknowns have a budget, as nodes do. Past it, the graph records how many unknowns of each family and reason were left out, and lists the budget under `coverage.limits`. Then the result size follows the configured ceilings and not the source length. ### Impact and scope Any large or minified bundle. CLI users get a result tens of times larger than the input (76x for the 1.66 MB file). Agents reading the CLI output, and anything that keeps the complete Evidence, pay for the unknowns in time, memory and bytes. `--max-semantic-nodes` and `--max-semantic-nodes-per-file` do not reduce them: with both at 1,000, the 10,000-statement file keeps 1,000 nodes and all 40,000 unknowns. ### REA version `rea-agents@6.5.0` (npm), and main at 2cf9c28f ### Runtime and platform Node 24.18.0, Fedora 44 x86_64 ### Target and artifact context Generated JavaScript (recipe above). Real-app figures come from the Flathub packages of Discord and Teams for Linux, and from Antigravity's Linux install. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Unknown reasons on the 40,000-statement file: ```text 80000 ambiguous-target object-flow 40000 dynamic-call call-flow 40000 dynamic-property object-flow ``` The same file on 6.5.0: 126,545,939 bytes, 20,000 nodes, 160,000 unknowns, 69.5 s, 1.24 GiB peak RSS. I also ran it on open PR #2092 (head b6a097b3), which replaces the default node ceilings. That run kept 280,001 nodes and the same 160,000 unknowns: 353,160,381 bytes, 160 s, 2.7 GiB peak RSS. On Discord, #2092 kept 317 unknowns, but 190,560 nodes and 249 MB. So #2092 does not bound this case. No reference in `unknowns` dangles: every `evidence.context_id`, `node_id` and `candidate_node_ids` entry resolves, at 1,000 and at 40,000 statements. ### Typed outcome or recovery expected A completed result with `coverage.truncated: true`, a named unknowns limit in `coverage.limits`, and per-family omitted counts for the dropped unknowns. ### Analysis or suspected cause `adoptFileProjection` in `src/application/javascript/JavaScriptSemanticGraphBuilder.ts` (lines 334 to 355) checks new nodes against `min(fileNodeCeiling, nodeCeiling - state.nodes.size)` and throws if they exceed it. The loop just below adopts every `projection.unknowns` entry with no budget. `recordUnavailableFile` (lines 388 to 405) and the getter-expansion path (lines 570 to 592) add unknowns without one too. I have not looked at where the projection decides to emit an unknown for a statement whose nodes were dropped. ### Additional context #1065 says complete Evidence keeps its unknowns, so if keeping all of them is intended, a page or a count-only view for unknowns would work instead. I can send a PR for whichever shape you prefer.

bsakaguchi closed this issue

2026-10-17 21:05:43 +00:00

bsakaguchi commented

2026-10-17 21:05:45 +00:00

Author

Owner

@morluto commented on GitHub (Oct 17, 2026):

Fixed on main in f4322b15e.

Unknown admission now happens at one producer owner before canonical IDs and provenance are interned. Every projection path uses it, including unavailable sources and getter-expansion frontiers. Independent defaults are 100000 unknowns/application and 20000/source; callers can select max_semantic_unknowns / max_semantic_unknowns_per_file (CLI flags use hyphens). A typed capacity offer travels from the parent to its owned worker. The streamed decoder and graph accumulator independently reject excess rows.

Coverage is partial/truncated when unknown retention drops emissions, names both unknown ceilings, and includes bounded omitted_unknown_occurrences rows by family/reason. These count examined emission occurrences, including repeats—not distinct discarded identities or frontiers in unexamined source. No discarded-ID set is retained. Raw source and existing node/execution/memory coverage remain authoritative. The node-only file truncation flag was renamed to distinguish it from overall graph truncation. Omission-free graph shape and identity remain unchanged.

Executed checks: 89 focused checks across ten files passed, including real owned-worker lifecycle controls, authenticated transfer over-allocation, direct/transferred equivalence, zero remaining capacity, graph traversal, CLI/stdio MCP exact Evidence/result parity and SDK-converted schema validity. A final three-file-capacity control rerun passed. Build, typecheck/lint, docs checks (five README checks), commit and push hooks passed. Two independent read-only source reviews found no blockers.

Before/after actual CLI control: four inert dynamic statements with a one-node ceiling retained 16 unknowns on baseline 926e18b3f. With the added two-unknown offer, the fix retains two and reports 14 omitted emission occurrences. Two-file CLI/MCP control retains three and reports 29 omitted occurrences, with matching Evidence identity across adapters.

18 completed small CLI cost tasks compared ordinary, unrestricted dynamic and explicitly bounded dynamic sources; first processes were separated from two alternating repeats. Ordinary/unrestricted replies, including Evidence IDs, were byte-identical. Bounded reply estimate: 8813→5492 cl100k tokens, intentionally reduced retained detail. Repeat medians for that task: elapsed 1.213→1.226s, CPU 1.262→1.269s, sampled process-family peak 524.1→522.0 MB. No speedup, true-cold or equivalent-retained-coverage claim; whole-agent/catalog/MCP and large-input costs unmeasured.

The reported 40000-statement stress run and existing heavy builder fixtures were not executed. All sources used here were tiny and inert; no investigated target execution or heavy profiling. This verifies the capacity mechanism, not real-app peak memory or upstream IR boundedness. Context7's Zod optionality docs informed omission preservation, checked against installed Zod 4.4.3 and actual schema/client tests; DeepWiki's compiler diagnostic analogy did not establish a resource bound and was not used as proof.

@morluto commented on GitHub (Oct 17, 2026): Fixed on main in f4322b15e. Unknown admission now happens at one producer owner before canonical IDs and provenance are interned. Every projection path uses it, including unavailable sources and getter-expansion frontiers. Independent defaults are 100000 unknowns/application and 20000/source; callers can select `max_semantic_unknowns` / `max_semantic_unknowns_per_file` (CLI flags use hyphens). A typed capacity offer travels from the parent to its owned worker. The streamed decoder and graph accumulator independently reject excess rows. Coverage is partial/truncated when unknown retention drops emissions, names both unknown ceilings, and includes bounded `omitted_unknown_occurrences` rows by family/reason. These count examined emission occurrences, including repeats—not distinct discarded identities or frontiers in unexamined source. No discarded-ID set is retained. Raw source and existing node/execution/memory coverage remain authoritative. The node-only file truncation flag was renamed to distinguish it from overall graph truncation. Omission-free graph shape and identity remain unchanged. Executed checks: 89 focused checks across ten files passed, including real owned-worker lifecycle controls, authenticated transfer over-allocation, direct/transferred equivalence, zero remaining capacity, graph traversal, CLI/stdio MCP exact Evidence/result parity and SDK-converted schema validity. A final three-file-capacity control rerun passed. Build, typecheck/lint, docs checks (five README checks), commit and push hooks passed. Two independent read-only source reviews found no blockers. Before/after actual CLI control: four inert dynamic statements with a one-node ceiling retained 16 unknowns on baseline 926e18b3f. With the added two-unknown offer, the fix retains two and reports 14 omitted emission occurrences. Two-file CLI/MCP control retains three and reports 29 omitted occurrences, with matching Evidence identity across adapters. 18 completed small CLI cost tasks compared ordinary, unrestricted dynamic and explicitly bounded dynamic sources; first processes were separated from two alternating repeats. Ordinary/unrestricted replies, including Evidence IDs, were byte-identical. Bounded reply estimate: 8813→5492 cl100k tokens, intentionally reduced retained detail. Repeat medians for that task: elapsed 1.213→1.226s, CPU 1.262→1.269s, sampled process-family peak 524.1→522.0 MB. No speedup, true-cold or equivalent-retained-coverage claim; whole-agent/catalog/MCP and large-input costs unmeasured. The reported 40000-statement stress run and existing heavy builder fixtures were not executed. All sources used here were tiny and inert; no investigated target execution or heavy profiling. This verifies the capacity mechanism, not real-app peak memory or upstream IR boundedness. Context7's Zod optionality docs informed omission preservation, checked against installed Zod 4.4.3 and actual schema/client tests; DeepWiki's compiler diagnostic analogy did not establish a resource bound and was not used as proof.

bsakaguchi referenced this issue

2026-10-31 01:05:21 +00:00

[PR #2494] [MERGED] fix(domain): preserve retryable capability failures through redaction #5273

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#2494

No description provided.

Rows
Columns