morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6635] [Bug] Environment state templates rewrite tokens embedded in astral letters or digits #4017

New issue

Closed

opened 2026-10-31 00:46:04 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:46:04 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6635

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The environment-state rule producer derives a label template from {id: 42, label: "𐐀42𐐁"} and then changes that label to "𐐀43𐐁" when generating ID 43. The characters immediately surrounding 42 are Deseret letters, so the number is embedded in a word rather than delimited.

The same false template occurs with supplementary-plane digits ("𝟙42𝟚"). ASCII-letter and BMP-letter controls preserve their original labels. An emoji control and a path control correctly permit substitution. These are observed calls to the production domain rule producer and application helpers; public CLI/MCP and an environment server were not run.

Reproduction

With the locked dependencies installed, run the following probe from the source checkout. My execution used Node's TypeScript transform with a resolver for the repository's .js-to-.ts source imports. The resolver changes module resolution only.

Create resolve.mjs in a disposable directory:

import { registerHooks } from 'node:module';
import { existsSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
registerHooks({
  resolve(specifier, context, nextResolve) {
    try { return nextResolve(specifier, context); }
    catch (error) {
      if (error.code === 'ERR_MODULE_NOT_FOUND' &&
          specifier.endsWith('.js') && /^(\.\/|\.\.\/)/.test(specifier)) {
        const candidate = new URL(specifier.slice(0, -3) + '.ts', context.parentURL);
        if (existsSync(fileURLToPath(candidate)))
          return nextResolve(candidate.href, context);
      }
      throw error;
    }
  }
});

Create probe.mjs beside it:

import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const source = p => pathToFileURL(resolve(
  process.argv[2], 'src/domain/analysisView', p)).href;
const { delimitedOffsets } = await import(source('environmentStateWriteValues.ts'));
const { explainCreate } = await import(source('environmentStateWriteExplain.ts'));
const { createdFields } = await import(source('environmentStateWriteApply.ts'));
for (const label of ['A42B', 'é42é', '𐐀42𐐁', '𝟙42𝟚', '😀42😀', '/items/42']) {
  const entity = { id: 42, label };
  const call = { path: '/items', pathValues: new Map(), body: {} };
  const explained = explainCreate({
    entity, call, idField: 'id', startMs: undefined, collection: [], type: []
  });
  const applied = createdFields(explained, entity, call,
    rule => ({ ok: true, value: rule.field === 'id' ? 43 : undefined }));
  console.log(JSON.stringify({
    label, offsets: delimitedOffsets(label, '42'),
    rule: explained.rules.find(x => x.field === 'label') ?? null,
    created_label: applied.ok ? applied.value.label : applied.error
  }));
}

Run, with work pointing to that disposable directory:

node --experimental-transform-types --import "$work/resolve.mjs" "$work/probe.mjs" "$PWD"

The saved executed probe additionally asserted successful application and generated ID 43 for every case. Each of the six cases ran once before drafting.

Expected behavior

Use complete Unicode characters when deciding whether a neighboring character is a letter or digit. Neither "𐐀42𐐁" nor "𝟙42𝟚" should produce a delimited occurrence or an ID-derived label template; both labels should stay unchanged when only the generated ID changes.

The delimiter helper's contract explicitly requires that neither neighbor be a letter or digit and uses Unicode letter/number properties. The existing tests already exclude a BMP letter next to the token.

Impact and scope

A bounded state-generation correctness bug: labels containing these supplementary-plane letters or digits can be falsely classified as templates and rewritten. The original input is not modified by this probe. Two affected labels and four controls were exercised; real captured applications, served environments, and frequency in existing data were not measured.

REA version

Source main revision a281455f0b83c088ea30ebbb887398888ed167fb, with package.json version 6.7.0 and locked dependencies. This reports the inspected source tree, including unreleased environment-state code; it does not establish that the published 6.7.0 package contains this code.

Runtime and platform

Node.js v24.22.1; Linux x86_64. Native TypeScript transformation and a source-extension resolver; no external analysis provider.

Target and artifact context

Authored ordinary JSON entities containing an integer ID and a Unicode string label. No binary target, captured private application data, or network access is needed.

Interface

Other

Production domain rule production and application.

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Reduced observed output:

{"label":"A42B","actual_offsets":[],"derived_label_rule":null,"created_label":"A42B"}
{"label":"é42é","actual_offsets":[],"derived_label_rule":null,"created_label":"é42é"}
{"label":"𐐀42𐐁","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"𐐀43𐐁"}
{"label":"𝟙42𝟚","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"𝟙43𝟚"}
{"label":"😀42😀","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"😀43😀"}
{"label":"/items/42","actual_offsets":[7],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"/items/43"}

The first two cases contain neighboring letters and remain constant. The next two contain neighboring supplementary-plane letters or digits and incorrectly change. The final two use actual delimiters and correctly change.

Typed outcome or recovery expected

For the affected labels, return no template rule and retain the original label while generating the new identity. Continue substituting truly delimited occurrences.

Analysis or suspected cause

Source-derived: string indexing in delimitedOffsets passes one UTF-16 code unit from either neighboring surrogate pair to isAlphanumeric. That isolated surrogate does not match a Unicode letter or number. Template discovery uses those offsets to derive a rule; the same delimiter helper then applies the false substitution.

Additional context

No source fix or restored behavior was tested. The bug was not rerun after its draft was written. Scope is ordinary data interpretation and state generation.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Two valid Unicode labels received an unsupported template rule and changed during production rule application; affected inputs are bounded.
Workaround status: unverified
Workaround basis: No workaround preserving the original Unicode text and state-generation behavior was evaluated.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6635 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The environment-state rule producer derives a label template from `{id: 42, label: "𐐀42𐐁"}` and then changes that label to `"𐐀43𐐁"` when generating ID 43. The characters immediately surrounding `42` are Deseret letters, so the number is embedded in a word rather than delimited. The same false template occurs with supplementary-plane digits (`"𝟙42𝟚"`). ASCII-letter and BMP-letter controls preserve their original labels. An emoji control and a path control correctly permit substitution. These are observed calls to the production domain rule producer and application helpers; public CLI/MCP and an environment server were not run. ## Reproduction With the locked dependencies installed, run the following probe from the source checkout. My execution used Node's TypeScript transform with a resolver for the repository's `.js`-to-`.ts` source imports. The resolver changes module resolution only. Create `resolve.mjs` in a disposable directory: ```js import { registerHooks } from 'node:module'; import { existsSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; registerHooks({ resolve(specifier, context, nextResolve) { try { return nextResolve(specifier, context); } catch (error) { if (error.code === 'ERR_MODULE_NOT_FOUND' && specifier.endsWith('.js') && /^(\.\/|\.\.\/)/.test(specifier)) { const candidate = new URL(specifier.slice(0, -3) + '.ts', context.parentURL); if (existsSync(fileURLToPath(candidate))) return nextResolve(candidate.href, context); } throw error; } } }); ``` Create `probe.mjs` beside it: ```js import { resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; const source = p => pathToFileURL(resolve( process.argv[2], 'src/domain/analysisView', p)).href; const { delimitedOffsets } = await import(source('environmentStateWriteValues.ts')); const { explainCreate } = await import(source('environmentStateWriteExplain.ts')); const { createdFields } = await import(source('environmentStateWriteApply.ts')); for (const label of ['A42B', 'é42é', '𐐀42𐐁', '𝟙42𝟚', '😀42😀', '/items/42']) { const entity = { id: 42, label }; const call = { path: '/items', pathValues: new Map(), body: {} }; const explained = explainCreate({ entity, call, idField: 'id', startMs: undefined, collection: [], type: [] }); const applied = createdFields(explained, entity, call, rule => ({ ok: true, value: rule.field === 'id' ? 43 : undefined })); console.log(JSON.stringify({ label, offsets: delimitedOffsets(label, '42'), rule: explained.rules.find(x => x.field === 'label') ?? null, created_label: applied.ok ? applied.value.label : applied.error })); } ``` Run, with `work` pointing to that disposable directory: ```sh node --experimental-transform-types --import "$work/resolve.mjs" "$work/probe.mjs" "$PWD" ``` The saved executed probe additionally asserted successful application and generated ID 43 for every case. Each of the six cases ran once before drafting. ## Expected behavior Use complete Unicode characters when deciding whether a neighboring character is a letter or digit. Neither `"𐐀42𐐁"` nor `"𝟙42𝟚"` should produce a delimited occurrence or an ID-derived label template; both labels should stay unchanged when only the generated ID changes. The [delimiter helper's contract](https://github.com/morluto/rea/blob/a281455f0b83c088ea30ebbb887398888ed167fb/src/domain/analysisView/environmentStateWriteValues.ts#L256-L286) explicitly requires that neither neighbor be a letter or digit and uses Unicode letter/number properties. The [existing tests](https://github.com/morluto/rea/blob/a281455f0b83c088ea30ebbb887398888ed167fb/src/domain/analysisView/environmentStateWriteValues.test.ts#L172-L194) already exclude a BMP letter next to the token. ## Impact and scope A bounded state-generation correctness bug: labels containing these supplementary-plane letters or digits can be falsely classified as templates and rewritten. The original input is not modified by this probe. Two affected labels and four controls were exercised; real captured applications, served environments, and frequency in existing data were not measured. ## REA version Source `main` revision `a281455f0b83c088ea30ebbb887398888ed167fb`, with package.json version 6.7.0 and locked dependencies. This reports the inspected source tree, including unreleased environment-state code; it does not establish that the published 6.7.0 package contains this code. ## Runtime and platform Node.js v24.22.1; Linux x86_64. Native TypeScript transformation and a source-extension resolver; no external analysis provider. ## Target and artifact context Authored ordinary JSON entities containing an integer ID and a Unicode string label. No binary target, captured private application data, or network access is needed. ## Interface Other Production domain rule production and application. ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Reduced observed output: ```json {"label":"A42B","actual_offsets":[],"derived_label_rule":null,"created_label":"A42B"} {"label":"é42é","actual_offsets":[],"derived_label_rule":null,"created_label":"é42é"} {"label":"𐐀42𐐁","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"𐐀43𐐁"} {"label":"𝟙42𝟚","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"𝟙43𝟚"} {"label":"😀42😀","actual_offsets":[2],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"😀43😀"} {"label":"/items/42","actual_offsets":[7],"derived_label_rule":{"field":"label","rule":"template","from":"id"},"created_label":"/items/43"} ``` The first two cases contain neighboring letters and remain constant. The next two contain neighboring supplementary-plane letters or digits and incorrectly change. The final two use actual delimiters and correctly change. ## Typed outcome or recovery expected For the affected labels, return no template rule and retain the original label while generating the new identity. Continue substituting truly delimited occurrences. ## Analysis or suspected cause Source-derived: string indexing in `delimitedOffsets` passes one UTF-16 code unit from either neighboring surrogate pair to `isAlphanumeric`. That isolated surrogate does not match a Unicode letter or number. [Template discovery](https://github.com/morluto/rea/blob/a281455f0b83c088ea30ebbb887398888ed167fb/src/domain/analysisView/environmentStateWriteCopies.ts#L14-L34) uses those offsets to derive a rule; the same delimiter helper then applies the false substitution. ## Additional context No source fix or restored behavior was tested. The bug was not rerun after its draft was written. Scope is ordinary data interpretation and state generation. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Two valid Unicode labels received an unsupported template rule and changed during production rule application; affected inputs are bounded. - Workaround status: unverified - Workaround basis: No workaround preserving the original Unicode text and state-generation behavior was evaluated.

bsakaguchi closed this issue

2026-10-31 00:46:05 +00:00

bsakaguchi commented

2026-10-31 00:46:05 +00:00

Author

Owner

@gongfpp commented on GitHub (Oct 23, 2026):

I plan to work on this existing-boundary correctness fix. I reproduced the UTF-16 code-unit boundary problem and will keep token offsets in UTF-16 while checking complete Unicode neighbors at the shared delimiter producer. Planned coverage includes astral letters/digits on either side, BMP/ASCII controls, emoji/path delimiters, overlapping matches, and a real served MCP create/read regression. No new provider, target execution authority, or schema changes are proposed. Please let me know if someone is already working on this or a maintainer intends to take it; I will coordinate rather than duplicate work. This comment is coordination, not an assignment or a scope/merge approval.

@gongfpp commented on GitHub (Oct 23, 2026): I plan to work on this existing-boundary correctness fix. I reproduced the UTF-16 code-unit boundary problem and will keep token offsets in UTF-16 while checking complete Unicode neighbors at the shared delimiter producer. Planned coverage includes astral letters/digits on either side, BMP/ASCII controls, emoji/path delimiters, overlapping matches, and a real served MCP create/read regression. No new provider, target execution authority, or schema changes are proposed. Please let me know if someone is already working on this or a maintainer intends to take it; I will coordinate rather than duplicate work. This comment is coordination, not an assignment or a scope/merge approval.

bsakaguchi referenced this issue

2026-10-31 01:11:01 +00:00

[PR #4017] [CLOSED] fix(reference-source): honor Node require extension order #5992

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4017

No description provided.

Rows
Columns