morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5954] [Bug] Web runtime retention-budget refusals return unreadable_output and doctor advice #3622

New issue

Closed

opened 2026-10-31 00:42:28 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:28 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5954

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

Executed the shared observe_web_execution application workflow against the repository’s simulated HTTP/WebSocket CDP browser. Three well-formed, oversized evidence cases hit the documented script-metadata, request-metadata and complete-source bounds. All returned unreadable_output / execution_failure with rea doctor remediation, without resource or reported_limits fields. A normal small-page control succeeded. These were controlled producer-boundary runs; no real browser was used.

Reproduction

From the source revision below with locked dependencies installed, save this test as tests/boundary/browser/repro.test.ts and run:

node node_modules/vitest/vitest.mjs run tests/boundary/browser/repro.test.ts --maxWorkers 1

The following is the executed test body with unrelated cases and unused imports removed. It keeps the published limits unchanged, supplies nine script metadata records with large source-map declarations, sends one oversized initiator after actual arming, and returns two 17 MiB sources below the protocol-message limit. Its assertions characterize the observed incorrect classification.

import { expect, it, onTestFinished } from 'vitest';
import { WebRuntimeService } from '../../../src/application/WebRuntimeService.js';
import { CdpWebRuntimeProvider } from '../../../src/browser/execution/CdpWebRuntimeProvider.js';
import { observeWebExecutionInputSchema } from '../../../src/domain/webExecution.js';
import { projectAnalysisError } from '../../../src/domain/analysisErrorProjection.js';
import { startRuntimeBrowser } from '../../fixtures/webRuntime.js';

it.each(['metadata', 'request', 'source', 'control'] as const)('runtime budget %s', async kind => {
  const browser = await startRuntimeBrowser({
    commandEvents: (command, origin) => kind === 'metadata' && command.method === 'Debugger.enable'
      ? Array.from({ length: 9 }, (_, index) => ({ method: 'Debugger.scriptParsed', sessionId: command.sessionId, params: {
          scriptId: `large-script-${index}`, url: `${origin}/large-${index}.js`, executionContextId: 1,
          startLine: 0, startColumn: 0, endLine: 0, endColumn: 1,
          hash: '0123456789abcdef'.repeat(4),
          sourceMapURL: 'data:application/json;base64,' + 'A'.repeat(1024 * 1024),
        } })) : undefined,
    commandResult: command => kind === 'source' && command.method === 'Debugger.getScriptSource'
      ? { scriptSource: ' '.repeat(17 * 1024 * 1024) } : undefined,
  });
  onTestFinished(() => browser.close());
  const provider = new CdpWebRuntimeProvider();
  const input = observeWebExecutionInputSchema.parse({ cdp_endpoint: browser.endpoint, target_id: 'allowed-page', observation_ms: kind === 'request' ? 1000 : 5 });
  const result = await new WebRuntimeService(provider).observe(input, {
    progress: { report: async update => {
      if (kind === 'request' && update.completed === 1) browser.emitEvent({
        method: 'Network.requestWillBeSent', sessionId: 'session-1', params: {
          requestId: 'large-request', frameId: 'runtime-main', timestamp: 12.5,
          request: { url: `${browser.endpoint}/request`, method: 'GET' },
          initiator: { type: 'script', stack: { callFrames: [], description: 'a'.repeat(8 * 1024 * 1024) } },
        },
      });
    } },
  });
  if (kind === 'control') { expect(result.ok).toBe(true); return; }
  if (result.ok) throw new Error('Expected bounded failure');
  const p = projectAnalysisError(result.error);
  expect(p.code).toBe('unreadable_output');
  expect(p.details).not.toHaveProperty('resource');
  expect(p.details).not.toHaveProperty('reported_limits');
  expect(p.remediation.action).toContain('rea doctor');
  expect(browser.commands.some(c => c.method === 'Target.detachFromTarget')).toBe(true);
  console.log('BUDGET', kind, JSON.stringify(p));
});

Expected behavior

Keep the published bounds and no-partial-success behavior, but classify an exceeded retention bound as a resource constraint and retain its effective numerical limit. Recovery should identify the fixed retention allowance, rather than imply that provider output is unreadable or that doctor can change the allowance.

The runtime bounds document these budgets. The resource error contract covers bounded admission refusals; AnalysisOutputError describes output parsing failures. The runtime guide does not prescribe a particular resource-kind enum.

Impact and scope

P2: callers lose the machine-readable distinction between capacity refusal and malformed provider output, and receive irrelevant recovery advice. The specific numerical budget still survives in details.reason, so a caller can inspect that text as a workaround. The rejection itself remains effective and no partial successful capture was returned. All three synthetic oversized cases reproduced; the control succeeded. Natural frequency on large real pages, other platforms and downstream automation behavior were not measured.

REA version

rea-agents 6.7.0; source revision 5241597594

Runtime and platform

macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3 with locked dependencies. Repository Node pin 24.18.0 was not exercised. Simulated CDP producer only.

Target and artifact context

Selected synthetic page target on a loopback CDP endpoint. Nine large script metadata events, an armed request with a large stack description, and two 17 MiB source strings; no external page or private artifact.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

The focused workflow run passed six cases, including four runtime cases and two independent export cases. Selected runtime output:

{"case":"metadata","code":"unreadable_output","category":"execution_failure","reason":"Script metadata exceeds the complete 8 MiB event budget."}
{"case":"request","code":"unreadable_output","category":"execution_failure","reason":"Request initiator evidence exceeds its complete 8 MiB event budget."}
{"case":"source","code":"unreadable_output","category":"execution_failure","reason":"Complete runtime source evidence exceeds its 32 MiB budget; no partial evidence is returned."}

Each full projection omitted resource and reported_limits and supplied: “Analysis returned an unreadable result. Run rea doctor and review the reported details; repeating the unchanged request is not expected to succeed.” All three failure cases sent Target.detachFromTarget. The small-capture control returned success.

Typed outcome or recovery expected

resource_constraint with the exhausted budget and effective byte limit in typed details, deterministic retryability and recovery that describes this fixed allowance. Malformed producer payloads should continue to be output failures. No limit increase or partial-success behavior is requested.

Analysis or suspected cause

Source-derived: metadata ingest catches its deliberate budget Error as AnalysisOutputError; source capture constructs that class directly. request admission does the same inside its catch-all. The shared projection is following that selected class. Public reachability is source-derived from MCP registration and CLI registration, both using the executed WebRuntimeService workflow.

Additional context

Fetched origin/main at ce1f4b1bba; the relevant collector code and class mapping were unchanged. Related #5631 concerns producers already returning resource_constraint with host-memory recovery; #5846 and #5848 concern firmware and EVM producers. Those repairs do not change these runtime collector branches. No regression range or real-browser failure frequency was established.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5954 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior Executed the shared observe_web_execution application workflow against the repository’s simulated HTTP/WebSocket CDP browser. Three well-formed, oversized evidence cases hit the documented script-metadata, request-metadata and complete-source bounds. All returned unreadable_output / execution_failure with rea doctor remediation, without resource or reported_limits fields. A normal small-page control succeeded. These were controlled producer-boundary runs; no real browser was used. ### Reproduction From the source revision below with locked dependencies installed, save this test as tests/boundary/browser/repro.test.ts and run: ```sh node node_modules/vitest/vitest.mjs run tests/boundary/browser/repro.test.ts --maxWorkers 1 ``` The following is the executed test body with unrelated cases and unused imports removed. It keeps the published limits unchanged, supplies nine script metadata records with large source-map declarations, sends one oversized initiator after actual arming, and returns two 17 MiB sources below the protocol-message limit. Its assertions characterize the observed incorrect classification. ```ts import { expect, it, onTestFinished } from 'vitest'; import { WebRuntimeService } from '../../../src/application/WebRuntimeService.js'; import { CdpWebRuntimeProvider } from '../../../src/browser/execution/CdpWebRuntimeProvider.js'; import { observeWebExecutionInputSchema } from '../../../src/domain/webExecution.js'; import { projectAnalysisError } from '../../../src/domain/analysisErrorProjection.js'; import { startRuntimeBrowser } from '../../fixtures/webRuntime.js'; it.each(['metadata', 'request', 'source', 'control'] as const)('runtime budget %s', async kind => { const browser = await startRuntimeBrowser({ commandEvents: (command, origin) => kind === 'metadata' && command.method === 'Debugger.enable' ? Array.from({ length: 9 }, (_, index) => ({ method: 'Debugger.scriptParsed', sessionId: command.sessionId, params: { scriptId: `large-script-${index}`, url: `${origin}/large-${index}.js`, executionContextId: 1, startLine: 0, startColumn: 0, endLine: 0, endColumn: 1, hash: '0123456789abcdef'.repeat(4), sourceMapURL: 'data:application/json;base64,' + 'A'.repeat(1024 * 1024), } })) : undefined, commandResult: command => kind === 'source' && command.method === 'Debugger.getScriptSource' ? { scriptSource: ' '.repeat(17 * 1024 * 1024) } : undefined, }); onTestFinished(() => browser.close()); const provider = new CdpWebRuntimeProvider(); const input = observeWebExecutionInputSchema.parse({ cdp_endpoint: browser.endpoint, target_id: 'allowed-page', observation_ms: kind === 'request' ? 1000 : 5 }); const result = await new WebRuntimeService(provider).observe(input, { progress: { report: async update => { if (kind === 'request' && update.completed === 1) browser.emitEvent({ method: 'Network.requestWillBeSent', sessionId: 'session-1', params: { requestId: 'large-request', frameId: 'runtime-main', timestamp: 12.5, request: { url: `${browser.endpoint}/request`, method: 'GET' }, initiator: { type: 'script', stack: { callFrames: [], description: 'a'.repeat(8 * 1024 * 1024) } }, }, }); } }, }); if (kind === 'control') { expect(result.ok).toBe(true); return; } if (result.ok) throw new Error('Expected bounded failure'); const p = projectAnalysisError(result.error); expect(p.code).toBe('unreadable_output'); expect(p.details).not.toHaveProperty('resource'); expect(p.details).not.toHaveProperty('reported_limits'); expect(p.remediation.action).toContain('rea doctor'); expect(browser.commands.some(c => c.method === 'Target.detachFromTarget')).toBe(true); console.log('BUDGET', kind, JSON.stringify(p)); }); ``` ### Expected behavior Keep the published bounds and no-partial-success behavior, but classify an exceeded retention bound as a resource constraint and retain its effective numerical limit. Recovery should identify the fixed retention allowance, rather than imply that provider output is unreadable or that doctor can change the allowance. The [runtime bounds](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/web-runtime.md#L51-L55) document these budgets. The [resource error contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisErrorCore.ts#L23-L55) covers bounded admission refusals; [AnalysisOutputError](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisErrorCore.ts#L105-L124) describes output parsing failures. The runtime guide does not prescribe a particular resource-kind enum. ### Impact and scope P2: callers lose the machine-readable distinction between capacity refusal and malformed provider output, and receive irrelevant recovery advice. The specific numerical budget still survives in details.reason, so a caller can inspect that text as a workaround. The rejection itself remains effective and no partial successful capture was returned. All three synthetic oversized cases reproduced; the control succeeded. Natural frequency on large real pages, other platforms and downstream automation behavior were not measured. ### REA version rea-agents 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e ### Runtime and platform macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3 with locked dependencies. Repository Node pin 24.18.0 was not exercised. Simulated CDP producer only. ### Target and artifact context Selected synthetic page target on a loopback CDP endpoint. Nine large script metadata events, an armed request with a large stack description, and two 17 MiB source strings; no external page or private artifact. ### Interface Other ### Earliest failure stage Capture or observation ### Relevant output or evidence The focused workflow run passed six cases, including four runtime cases and two independent export cases. Selected runtime output: ```json {"case":"metadata","code":"unreadable_output","category":"execution_failure","reason":"Script metadata exceeds the complete 8 MiB event budget."} {"case":"request","code":"unreadable_output","category":"execution_failure","reason":"Request initiator evidence exceeds its complete 8 MiB event budget."} {"case":"source","code":"unreadable_output","category":"execution_failure","reason":"Complete runtime source evidence exceeds its 32 MiB budget; no partial evidence is returned."} ``` Each full projection omitted resource and reported_limits and supplied: “Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.” All three failure cases sent Target.detachFromTarget. The small-capture control returned success. ### Typed outcome or recovery expected resource_constraint with the exhausted budget and effective byte limit in typed details, deterministic retryability and recovery that describes this fixed allowance. Malformed producer payloads should continue to be output failures. No limit increase or partial-success behavior is requested. ### Analysis or suspected cause Source-derived: [metadata ingest](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/execution/CdpRuntimeSources.ts#L43-L90) catches its deliberate budget Error as AnalysisOutputError; [source capture](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/execution/CdpRuntimeSources.ts#L264-L271) constructs that class directly. [request admission](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/execution/CdpRuntimeRequests.ts#L40-L44) does the same inside its [catch-all](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/execution/CdpRuntimeRequests.ts#L93-L98). The shared projection is following that selected class. Public reachability is source-derived from [MCP registration](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/server/registerWebRuntimeTools.ts#L21-L48) and [CLI registration](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/cli/webRuntimeCommands.ts#L30-L64), both using the executed WebRuntimeService workflow. ### Additional context Fetched origin/main at ce1f4b1bbaced57529d609f3007a76de583e11c4; the relevant collector code and class mapping were unchanged. Related #5631 concerns producers already returning resource_constraint with host-memory recovery; #5846 and #5848 concern firmware and EVM producers. Those repairs do not change these runtime collector branches. No regression range or real-browser failure frequency was established.

bsakaguchi closed this issue

2026-10-31 00:42:28 +00:00

bsakaguchi commented

2026-10-31 00:42:29 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked resolved on main at d87e73f2db.

I replayed the report's three oversized retention cases and small-page control through WebRuntimeService and CdpWebRuntimeProvider with a simulated HTTP/WebSocket CDP endpoint. All four restoration tests passed. Metadata and armed-request overflow now return non-retryable resource_constraint / capacity with an 8,388,608-byte allowance; the two complete 17 MiB source case reports its 33,554,432-byte allowance. Each reports measured retained bytes above its limit, fixed-capture recovery without doctor/unreadable advice, and failure rather than partial success. Every case detaches, and the small-page control returns successful Evidence. No real browser was exercised.

Relevant implementation: CdpRuntimeRetention.ts, CdpRuntimeSources.ts.

Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked resolved on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. I replayed the report's three oversized retention cases and small-page control through `WebRuntimeService` and `CdpWebRuntimeProvider` with a simulated HTTP/WebSocket CDP endpoint. All four restoration tests passed. Metadata and armed-request overflow now return non-retryable `resource_constraint / capacity` with an 8,388,608-byte allowance; the two complete 17 MiB source case reports its 33,554,432-byte allowance. Each reports measured retained bytes above its limit, fixed-capture recovery without doctor/unreadable advice, and failure rather than partial success. Every case detaches, and the small-page control returns successful Evidence. No real browser was exercised. Relevant implementation: [CdpRuntimeRetention.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/browser/execution/CdpRuntimeRetention.ts#L6), [CdpRuntimeSources.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/browser/execution/CdpRuntimeSources.ts#L69). Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:09:58 +00:00

[PR #3622] [CLOSED] feat(analysis-view): report value-free request header facets in API surfaces #5858

bsakaguchi referenced this issue

2026-11-03 00:30:16 +00:00

[GH-ISSUE #8626] [Bug] HTML artifact analysis silently omits executable SVG scripts #7500