morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6750] [Bug] Environment query search drops exact Greek sigma matches and cannot learn recorded searches #4108

New issue

Closed

opened 2026-10-31 00:46:53 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:46:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6750

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The environment state's case-insensitive substring search drops an exact substring already present in a field. A row named ΟΣ is excluded by q=Σ, although q=Ο and q=ΟΣ both select it. The same mismatch prevents the query learner from explaining a recorded result containing that row.

Observed directly through the production query functions. No CLI, MCP, exported environment, or live HTTP service was exercised.

Reproduction

From a checkout with its locked dependencies installed, save and run this script with Node and the repository's tsx loader:

import { evaluateQuery, lineChoices, learnQuery } from './src/domain/analysisView/environmentStateListQuery.ts';

const rows = [{ id: '1', name: 'ΟΣ' }, { id: '2', name: 'Alpha' }];
const entities = rows.map(fields => ({
  type: '/api/items', collection: '/api/items', id: fields.id,
  fields, view_fields: [], record_ordinals: [],
}));
const parameters = { q: { role: 'search', fields: ['name'] } };
for (const term of ['Ο', 'Σ', 'ΟΣ']) {
  const values = new Map([['q', term]]);
  const result = evaluateQuery(parameters, values, entities);
  const choices = lineChoices(values, [rows[0]]);
  const learned = learnQuery([{ ordinal: 0, values, listed: ['1'], choices, entities }]);
  console.log(JSON.stringify({ term, exactSubstring: rows[0].name.includes(term),
    page: result.page.map(x => x.id), total: result.total, choices,
    learned: learned ?? null }));
}

node --import tsx ./probe.mjs

One probe run produced the results below. The original probe used an equivalent relative import from its location outside the checkout.

Expected behavior

An exact substring should remain a match when matching ignores case. Searching Σ should retain row 1, report total 1, and allow the recorded search to be learned. The existing test contract explicitly models substring searches ignoring case.

Impact and scope

The demonstrated input returns an incorrect empty page and total, and loses an explainable recorded search. This affects Greek text where contextual lowercasing changes capital sigma to final sigma. Broader Unicode reach and complete exported-environment behavior were not tested. The control queries match the same row; a general workaround preserving arbitrary substring-search results has not been verified.

REA version

Source revision 2bcd7e6d13f0a8118d6619718254255e84f75d86 (package.json: 6.7.0).

Runtime and platform

Node.js v24.22.1; Linux x64; repository-locked dependencies installed with npm 11.16.0. No native analysis provider required.

Target and artifact context

Two inert in-memory entity records for /api/items; no external target.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Reduced output from the single probe run:

{"term":"Ο","exactSubstring":true,"page":["1"],"total":1,"learned":{"parameters":{"q":{"role":"search","fields":["name"]}},"record_ordinals":[0]}}
{"term":"Σ","exactSubstring":true,"page":[],"total":0,"choices":[["q",[{"role":"inert","values":["Σ"]}]]],"learned":null}
{"term":"ΟΣ","exactSubstring":true,"page":["1"],"total":1,"learned":{"parameters":{"q":{"role":"search","fields":["name"]}},"record_ordinals":[0]}}

Typed outcome or recovery expected

Preserve the matching row and correct total, and retain the supported search role when explaining the recording.

Analysis or suspected cause

Source-derived cause: query matching and search-field discovery lowercase the complete field and the standalone term separately. ΟΣ lowercases to ος, while standalone Σ lowercases to σ; the resulting substring comparison fails.

Additional context

Related matching failures exist in native search (#6634), signal discovery (#6714), and packet-field discovery (#6735). This report concerns the separate environment query evaluation and inference functions cited above.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A local production-boundary probe returned a false empty search result and could not learn the corresponding recorded search for an exact Greek substring.
Workaround status: unverified
Workaround basis: Whole-name and prefix controls return the demonstrated row, but no general workaround preserving arbitrary substring-search results was exercised.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6750 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The environment state's case-insensitive substring search drops an exact substring already present in a field. A row named `ΟΣ` is excluded by `q=Σ`, although `q=Ο` and `q=ΟΣ` both select it. The same mismatch prevents the query learner from explaining a recorded result containing that row. Observed directly through the production query functions. No CLI, MCP, exported environment, or live HTTP service was exercised. ### Reproduction From a checkout with its locked dependencies installed, save and run this script with Node and the repository's `tsx` loader: ```js import { evaluateQuery, lineChoices, learnQuery } from './src/domain/analysisView/environmentStateListQuery.ts'; const rows = [{ id: '1', name: 'ΟΣ' }, { id: '2', name: 'Alpha' }]; const entities = rows.map(fields => ({ type: '/api/items', collection: '/api/items', id: fields.id, fields, view_fields: [], record_ordinals: [], })); const parameters = { q: { role: 'search', fields: ['name'] } }; for (const term of ['Ο', 'Σ', 'ΟΣ']) { const values = new Map([['q', term]]); const result = evaluateQuery(parameters, values, entities); const choices = lineChoices(values, [rows[0]]); const learned = learnQuery([{ ordinal: 0, values, listed: ['1'], choices, entities }]); console.log(JSON.stringify({ term, exactSubstring: rows[0].name.includes(term), page: result.page.map(x => x.id), total: result.total, choices, learned: learned ?? null })); } ``` ```sh node --import tsx ./probe.mjs ``` One probe run produced the results below. The original probe used an equivalent relative import from its location outside the checkout. ### Expected behavior An exact substring should remain a match when matching ignores case. Searching `Σ` should retain row `1`, report total `1`, and allow the recorded search to be learned. The [existing test contract](https://github.com/morluto/rea/blob/2bcd7e6d13f0a8118d6619718254255e84f75d86/src/domain/analysisView/environmentStateListQuery.test.ts#L80) explicitly models substring searches ignoring case. ### Impact and scope The demonstrated input returns an incorrect empty page and total, and loses an explainable recorded search. This affects Greek text where contextual lowercasing changes capital sigma to final sigma. Broader Unicode reach and complete exported-environment behavior were not tested. The control queries match the same row; a general workaround preserving arbitrary substring-search results has not been verified. ### REA version Source revision `2bcd7e6d13f0a8118d6619718254255e84f75d86` (`package.json`: `6.7.0`). ### Runtime and platform Node.js `v24.22.1`; Linux x64; repository-locked dependencies installed with npm `11.16.0`. No native analysis provider required. ### Target and artifact context Two inert in-memory entity records for `/api/items`; no external target. ### Interface Other ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Reduced output from the single probe run: ```json {"term":"Ο","exactSubstring":true,"page":["1"],"total":1,"learned":{"parameters":{"q":{"role":"search","fields":["name"]}},"record_ordinals":[0]}} {"term":"Σ","exactSubstring":true,"page":[],"total":0,"choices":[["q",[{"role":"inert","values":["Σ"]}]]],"learned":null} {"term":"ΟΣ","exactSubstring":true,"page":["1"],"total":1,"learned":{"parameters":{"q":{"role":"search","fields":["name"]}},"record_ordinals":[0]}} ``` ### Typed outcome or recovery expected Preserve the matching row and correct total, and retain the supported search role when explaining the recording. ### Analysis or suspected cause Source-derived cause: [query matching](https://github.com/morluto/rea/blob/2bcd7e6d13f0a8118d6619718254255e84f75d86/src/domain/analysisView/environmentStateListQuery.ts#L108) and [search-field discovery](https://github.com/morluto/rea/blob/2bcd7e6d13f0a8118d6619718254255e84f75d86/src/domain/analysisView/environmentStateListQuery.ts#L370) lowercase the complete field and the standalone term separately. `ΟΣ` lowercases to `ος`, while standalone `Σ` lowercases to `σ`; the resulting substring comparison fails. ### Additional context Related matching failures exist in native search (#6634), signal discovery (#6714), and packet-field discovery (#6735). This report concerns the separate environment query evaluation and inference functions cited above. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A local production-boundary probe returned a false empty search result and could not learn the corresponding recorded search for an exact Greek substring. - Workaround status: unverified - Workaround basis: Whole-name and prefix controls return the demonstrated row, but no general workaround preserving arbitrary substring-search results was exercised.

bsakaguchi closed this issue

2026-10-31 00:46:53 +00:00

bsakaguchi commented

2026-10-31 00:46:54 +00:00

Author

Owner

@JingHao-Leon commented on GitHub (Oct 23, 2026):

Picking this up — the repro is fully offline (production query functions + tsx), which makes the normalization path straightforward to pin down. Suspecting the case-folding step maps final sigma (U+03C2) / capital sigma (U+03A3) inconsistently between the haystack and the needle, so an exact substring survives String.includes but not the folded comparison. Will confirm against environmentStateListQuery.ts and link the fix.

@JingHao-Leon commented on GitHub (Oct 23, 2026): Picking this up — the repro is fully offline (production query functions + tsx), which makes the normalization path straightforward to pin down. Suspecting the case-folding step maps final sigma (U+03C2) / capital sigma (U+03A3) inconsistently between the haystack and the needle, so an exact substring survives `String.includes` but not the folded comparison. Will confirm against `environmentStateListQuery.ts` and link the fix.

bsakaguchi commented

2026-10-31 00:46:54 +00:00

Author

Owner

@JingHao-Leon commented on GitHub (Oct 23, 2026):

Fix in PR #6791. Root cause confirmed and it is subtler than a missing case-fold: JavaScript's toLowerCase is context-sensitive for Greek — a capital sigma at the end of a word lowercases to the final form ς (U+03C2), while the same letter standalone lowercases to the medial σ (U+03C3). So the folded haystack for ΟΣ ends in ς and a folded needle σ never matches it, even though "ΟΣ".includes("Σ") is plainly true on the raw strings.

The fix folds both sides through a shared caseFold helper (lowercase, then collapse ς→σ) in contains and searchFields — position-independent, matching how users type the letter. Verified with the issue's probe: q=Σ now returns the row and the learner explains it; Ο and ΟΣ still match. Three regression tests added.

@JingHao-Leon commented on GitHub (Oct 23, 2026): Fix in PR #6791. Root cause confirmed and it is subtler than a missing case-fold: JavaScript's `toLowerCase` is context-sensitive for Greek — a capital sigma at the **end of a word** lowercases to the final form ς (U+03C2), while the same letter standalone lowercases to the medial σ (U+03C3). So the folded haystack for `ΟΣ` ends in ς and a folded needle `σ` never matches it, even though `"ΟΣ".includes("Σ")` is plainly true on the raw strings. The fix folds both sides through a shared `caseFold` helper (lowercase, then collapse ς→σ) in `contains` and `searchFields` — position-independent, matching how users type the letter. Verified with the issue's probe: `q=Σ` now returns the row and the learner explains it; `Ο` and `ΟΣ` still match. Three regression tests added.

bsakaguchi referenced this issue

2026-10-31 00:49:35 +00:00

[GH-ISSUE #7405] [Bug] docs/index.md repeats the native-inspection task with inconsistent guide lists #4412

bsakaguchi referenced this issue

2026-10-31 01:11:34 +00:00

[PR #4108] docs: route every guide through the sidebar and guide index #6063

bsakaguchi referenced this issue

2026-11-01 21:10:54 +00:00

[GH-ISSUE #3795] [Tracker] Open documentation issues and pull requests (October 2026) #1468