[GH-ISSUE #3878] [Bug] Passive browser captures drop the URL fragment from target, frame, and session URLs #1498

New issue

Closed

opened 2026-10-16 21:06:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:06:13 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3878

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

Passive browser captures report the page URL without its fragment. For a page at http://127.0.0.1:19416/page?mode=fixture#details, rea inspect-web-page, rea capture-web-screenshot, and rea observe-web-session all exit 0 but report target, frame, and session URLs without #details. Chrome returns the fragment separately as Page.Frame.urlFragment, and REA reads only Page.Frame.url.

All three commands exit 0 and report http://127.0.0.1:19416/page?mode=fixture without the fragment: inspect-web-page in target.url and frames[].url, capture-web-screenshot in target.url, and observe-web-session in target.initial_url and target.final_url. The browser's own discovery response for the same target still includes #details.

Reproduction

Save the script below as reproduce.mjs in an empty directory outside a REA source checkout. It uses the checkout's pinned playwright-core to launch its own headless browser and profile with remote debugging on loopback port 19415, serves a page on loopback port 19416, loads it with a fragment, and runs the three public CLI commands against that target.
Run node reproduce.mjs <path-to-rea-checkout> <path-to-chromium-executable>.

import { createServer } from 'node:http';
import { spawn } from 'node:child_process';
import { pathToFileURL } from 'node:url';
import { join, dirname } from 'node:path';
const [checkout, executable] = process.argv.slice(2);
const { chromium } = await import(pathToFileURL(join(checkout, 'node_modules/playwright-core/index.mjs')));
const server = createServer((request, response) => {
  response.setHeader('Content-Type', 'text/html');
  response.end('<title>fragment fixture</title><h1>Fragment fixture</h1>');
});
await new Promise(resolve => server.listen(19416, '127.0.0.1', resolve));
const context = await chromium.launchPersistentContext(join(dirname(new URL(import.meta.url).pathname), 'profile'), {
  executablePath: executable, headless: true,
  args: ['--remote-debugging-port=19415', '--disable-background-networking']
});
try {
  const page = context.pages()[0];
  await page.goto('http://127.0.0.1:19416/page?mode=fixture#details');
  console.log('browser:', context.browser().version());
  const cdp = await context.newCDPSession(page);
  const { frameTree: { frame } } = await cdp.send('Page.getFrameTree');
  console.log('raw frame:', JSON.stringify({url: frame.url, urlFragment: frame.urlFragment}));
  await cdp.detach();
  const targets = await (await fetch('http://127.0.0.1:19415/json/list')).json();
  const target = targets.find(target => target.type === 'page');
  console.log('discovery URL:', target.url);
  for (const command of ['inspect-web-page', 'capture-web-screenshot', 'observe-web-session']) {
    const args = [join(checkout, 'scripts/rea.mjs'), command, 'http://127.0.0.1:19415', target.id,
      ...(command === 'capture-web-screenshot' ? [] : ['--observation-ms', '10']), '--json'];
    const result = await new Promise((resolve, reject) => {
      const child = spawn(process.execPath, args);
      let stdout = '', stderr = '';
      child.stdout.on('data', data => stdout += data);
      child.stderr.on('data', data => stderr += data);
      child.on('error', reject);
      child.on('close', code => resolve({code, stdout, stderr}));
    });
    const value = JSON.parse(result.stdout).normalized_result;
    console.log(command, JSON.stringify({exit: result.code, url: value.target.url,
      initial_url: value.target.initial_url, final_url: value.target.final_url,
      frames: value.frames?.map(frame => frame.url)}));
  }
} finally {
  await context.close();
  server.closeAllConnections();
  await new Promise(resolve => server.close(resolve));
}

Expected behavior

Captured target.url, frames[].url, target.initial_url, and target.final_url keep the #details fragment. docs/browser-observation.md says ordinary fragments remain intact and that URL observations preserve the complete local URL, including fragments.

Impact and scope

Captures lose current document-location evidence, so hash-routed application states and anchor locations collapse to the same reported URL; screenshot pixels and DOM contents are not shown to be affected.

Workaround: available. The browser's HTTP discovery response (/json/list) still reports the full URL with the fragment for the same target, although this does not repair the stored capture.

Reporter's impact estimate: P2 (supported).

REA version

6.5.0, built from source at main commit 034342ba28 (latest npm release is 6.5.0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64; playwright-core 1.58.2 (repository-pinned) driving headless Chrome for Testing 153.0.8010.12

Target and artifact context

Headless Chromium page over CDP on loopback, URL with a fragment

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

Expected source: docs/browser-observation.md:247 (ordinary query values, fragments, and parameter order remain intact) and docs/browser-observation.md:279 "Data minimization" (URL observations preserve the complete local URL, including fragments)
Failure source: observed output of the three CLI commands from the script above, compared with the raw CDP frame and the HTTP discovery URL for the same target
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

browser: 153.0.8010.12
raw frame: {"url":"http://127.0.0.1:19416/page?mode=fixture","urlFragment":"#details"}
discovery URL: http://127.0.0.1:19416/page?mode=fixture#details
inspect-web-page {"exit":0,"url":"http://127.0.0.1:19416/page?mode=fixture","frames":["http://127.0.0.1:19416/page?mode=fixture"]}
capture-web-screenshot {"exit":0,"url":"http://127.0.0.1:19416/page?mode=fixture"}
observe-web-session {"exit":0,"initial_url":"http://127.0.0.1:19416/page?mode=fixture","final_url":"http://127.0.0.1:19416/page?mode=fixture"}

Source-derived reasoning: src/browser/CdpCaptureDocuments.ts:52 sanitizes only frame.url for frame inventories, and mainFrameUrl at line 83 returns only frame.url. That helper feeds src/browser/CdpPageCapture.ts:118 (projected into the target at line 340), src/browser/CdpScreenshot.ts:55 (projected at line 121), and src/browser/CdpObservationSession.ts:57 and 395-400 (initial and final URL). None of these adapters reads urlFragment.

Typed outcome or recovery expected

Captured target, frame, initial, and final URLs should include the page's fragment (combining Page.Frame.url with Page.Frame.urlFragment), as documented for URL observations.

Restoration check:

With the script above, the three CLI lines currently print URLs ending in ?mode=fixture; after a fix they print http://127.0.0.1:19416/page?mode=fixture#details in every reported target, frame, initial, and final URL. Control: loading http://127.0.0.1:19416/page?mode=fixture without a fragment continues to report that exact URL with no trailing #.

Analysis or suspected cause

Source-derived: src/browser/CdpCaptureDocuments.ts:52 and :83 read only Page.Frame.url; the helper feeds src/browser/CdpPageCapture.ts:118, src/browser/CdpScreenshot.ts:55, and src/browser/CdpObservationSession.ts:57 and 395-400; no adapter reads urlFragment.

Additional context

This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3878 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior Passive browser captures report the page URL without its fragment. For a page at `http://127.0.0.1:19416/page?mode=fixture#details`, `rea inspect-web-page`, `rea capture-web-screenshot`, and `rea observe-web-session` all exit 0 but report target, frame, and session URLs without `#details`. Chrome returns the fragment separately as `Page.Frame.urlFragment`, and REA reads only `Page.Frame.url`. All three commands exit 0 and report `http://127.0.0.1:19416/page?mode=fixture` without the fragment: `inspect-web-page` in `target.url` and `frames[].url`, `capture-web-screenshot` in `target.url`, and `observe-web-session` in `target.initial_url` and `target.final_url`. The browser's own discovery response for the same target still includes `#details`. ### Reproduction 1. Save the script below as `reproduce.mjs` in an empty directory outside a REA source checkout. It uses the checkout's pinned `playwright-core` to launch its own headless browser and profile with remote debugging on loopback port 19415, serves a page on loopback port 19416, loads it with a fragment, and runs the three public CLI commands against that target. 2. Run `node reproduce.mjs <path-to-rea-checkout> <path-to-chromium-executable>`. ```js import { createServer } from 'node:http'; import { spawn } from 'node:child_process'; import { pathToFileURL } from 'node:url'; import { join, dirname } from 'node:path'; const [checkout, executable] = process.argv.slice(2); const { chromium } = await import(pathToFileURL(join(checkout, 'node_modules/playwright-core/index.mjs'))); const server = createServer((request, response) => { response.setHeader('Content-Type', 'text/html'); response.end('<title>fragment fixture</title><h1>Fragment fixture</h1>'); }); await new Promise(resolve => server.listen(19416, '127.0.0.1', resolve)); const context = await chromium.launchPersistentContext(join(dirname(new URL(import.meta.url).pathname), 'profile'), { executablePath: executable, headless: true, args: ['--remote-debugging-port=19415', '--disable-background-networking'] }); try { const page = context.pages()[0]; await page.goto('http://127.0.0.1:19416/page?mode=fixture#details'); console.log('browser:', context.browser().version()); const cdp = await context.newCDPSession(page); const { frameTree: { frame } } = await cdp.send('Page.getFrameTree'); console.log('raw frame:', JSON.stringify({url: frame.url, urlFragment: frame.urlFragment})); await cdp.detach(); const targets = await (await fetch('http://127.0.0.1:19415/json/list')).json(); const target = targets.find(target => target.type === 'page'); console.log('discovery URL:', target.url); for (const command of ['inspect-web-page', 'capture-web-screenshot', 'observe-web-session']) { const args = [join(checkout, 'scripts/rea.mjs'), command, 'http://127.0.0.1:19415', target.id, ...(command === 'capture-web-screenshot' ? [] : ['--observation-ms', '10']), '--json']; const result = await new Promise((resolve, reject) => { const child = spawn(process.execPath, args); let stdout = '', stderr = ''; child.stdout.on('data', data => stdout += data); child.stderr.on('data', data => stderr += data); child.on('error', reject); child.on('close', code => resolve({code, stdout, stderr})); }); const value = JSON.parse(result.stdout).normalized_result; console.log(command, JSON.stringify({exit: result.code, url: value.target.url, initial_url: value.target.initial_url, final_url: value.target.final_url, frames: value.frames?.map(frame => frame.url)})); } } finally { await context.close(); server.closeAllConnections(); await new Promise(resolve => server.close(resolve)); } ``` ### Expected behavior Captured `target.url`, `frames[].url`, `target.initial_url`, and `target.final_url` keep the `#details` fragment. `docs/browser-observation.md` says ordinary fragments remain intact and that URL observations preserve the complete local URL, including fragments. ### Impact and scope Captures lose current document-location evidence, so hash-routed application states and anchor locations collapse to the same reported URL; screenshot pixels and DOM contents are not shown to be affected. Workaround: available. The browser's HTTP discovery response (`/json/list`) still reports the full URL with the fragment for the same target, although this does not repair the stored capture. Reporter's impact estimate: P2 (supported). ### REA version 6.5.0, built from source at main commit 034342ba28c8e56c3746be92b78aa7299afa02bd (latest npm release is 6.5.0) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64; playwright-core 1.58.2 (repository-pinned) driving headless Chrome for Testing 153.0.8010.12 ### Target and artifact context Headless Chromium page over CDP on loopback, URL with a fragment ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence - Expected source: docs/browser-observation.md:247 (ordinary query values, fragments, and parameter order remain intact) and docs/browser-observation.md:279 "Data minimization" (URL observations preserve the complete local URL, including fragments) - Failure source: observed output of the three CLI commands from the script above, compared with the raw CDP frame and the HTTP discovery URL for the same target - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete ```text browser: 153.0.8010.12 raw frame: {"url":"http://127.0.0.1:19416/page?mode=fixture","urlFragment":"#details"} discovery URL: http://127.0.0.1:19416/page?mode=fixture#details inspect-web-page {"exit":0,"url":"http://127.0.0.1:19416/page?mode=fixture","frames":["http://127.0.0.1:19416/page?mode=fixture"]} capture-web-screenshot {"exit":0,"url":"http://127.0.0.1:19416/page?mode=fixture"} observe-web-session {"exit":0,"initial_url":"http://127.0.0.1:19416/page?mode=fixture","final_url":"http://127.0.0.1:19416/page?mode=fixture"} ``` Source-derived reasoning: `src/browser/CdpCaptureDocuments.ts:52` sanitizes only `frame.url` for frame inventories, and `mainFrameUrl` at line 83 returns only `frame.url`. That helper feeds `src/browser/CdpPageCapture.ts:118` (projected into the target at line 340), `src/browser/CdpScreenshot.ts:55` (projected at line 121), and `src/browser/CdpObservationSession.ts:57` and 395-400 (initial and final URL). None of these adapters reads `urlFragment`. ### Typed outcome or recovery expected Captured target, frame, initial, and final URLs should include the page's fragment (combining Page.Frame.url with Page.Frame.urlFragment), as documented for URL observations. Restoration check: With the script above, the three CLI lines currently print URLs ending in `?mode=fixture`; after a fix they print `http://127.0.0.1:19416/page?mode=fixture#details` in every reported target, frame, initial, and final URL. Control: loading `http://127.0.0.1:19416/page?mode=fixture` without a fragment continues to report that exact URL with no trailing `#`. ### Analysis or suspected cause Source-derived: src/browser/CdpCaptureDocuments.ts:52 and :83 read only Page.Frame.url; the helper feeds src/browser/CdpPageCapture.ts:118, src/browser/CdpScreenshot.ts:55, and src/browser/CdpObservationSession.ts:57 and 395-400; no adapter reads urlFragment. ### Additional context This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

bsakaguchi referenced this issue

2026-10-17 21:11:59 +00:00

[PR #1513] [MERGED] fix: bind identity and coverage to the producing rule #2933

bsakaguchi closed this issue

2026-10-31 00:31:04 +00:00