morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6888] [Bug] SQLite UTF-16 schema inventories exhaust reply budget on internal hex keys #4222

New issue

Closed

opened 2026-10-31 00:47:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:50 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6888

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspectSqliteDatabaseSnapshot rejects valid UTF-16le and UTF-16be databases as exceeding the 16 MiB JSON reply budget, although the equivalent UTF-8 schema produces a 4,463,359-byte inventory worker reply or an 8,925,066-byte detailed-schema worker reply. Both UTF-16 fixtures passed SQLite PRAGMA integrity_check. The actual source worker also returned ok: false, reason: "output-limit" for the UTF-16le inventory.

The fixture contains 34 ordinary tables with long ASCII names and one TEXT column each. There are no rows, application functions, views, extensions, or custom collations.

Reproduction

From a checkout named rea, save this probe beside the checkout as sqlite-inventory-budget-probe.mjs. This is the fixture and production-boundary invocation used in the captured run (output reduced here):

import { DatabaseSync } from 'node:sqlite';
import { rmSync } from 'node:fs';
import { inspectSqliteDatabaseSnapshot } from './rea/src/sqlite/SqliteDatabaseInspection.ts';
import { jsonByteLength } from './rea/src/domain/jsonSerialization.ts';

const names = Array.from({ length: 34 }, (_, i) =>
  `${i}_${'x'.repeat(128 * 1024)}`);
for (const encoding of ['UTF-8', 'UTF-16le', 'UTF-16be']) {
  const path = new URL(`./sqlite-inventory-budget-${encoding}.db`, import.meta.url).pathname;
  rmSync(path, { force: true });
  const db = new DatabaseSync(path);
  db.exec(`PRAGMA encoding='${encoding}'; BEGIN;`);
  for (const name of names)
    db.exec(`CREATE TABLE "${name}"(value TEXT)`);
  db.exec('COMMIT');
  console.log(encoding, db.prepare('PRAGMA integrity_check').all());
  db.close();
  for (const schema_scope of ['inventory', 'database']) {
    try {
      const inspection = inspectSqliteDatabaseSnapshot(path, { path, schema_scope });
      console.log(encoding, schema_scope, inspection.schema.tables.length,
        jsonByteLength({ ok: true, inspection }));
    } catch (error) {
      console.log(encoding, schema_scope, error.reason, error.message);
    }
  }
}

Command used, expressed relative to the clone root:

mise exec node@24.22.1 -- node --import ./node_modules/tsx/dist/loader.mjs ../sqlite-inventory-budget-probe.mjs

Observed once per encoding/scope. UTF-8 succeeded in both scopes; both UTF-16 encodings failed in both scopes. A separate invocation of the clone's src/sqlite/SqliteDatabaseWorker.ts reproduced the UTF-16le inventory refusal in its JSON protocol.

Expected behavior

Apply the JSON reply allowance to the encoded returned facts. Internal metadata keys should not impose a smaller, encoding-dependent reply limit. Preserve the supported database encoding and complete schema rather than reporting an oversized reply for this fixture.

The documented compact inventory and detailed-schema scopes are both below the documented 16 MiB worker reply allowance here. The existing large-name schema regression also expects schemas with many long names to succeed when their complete JSON fits.

Impact and scope

Bounded compatibility failure for schemas with enough retained name bytes. This is an artificial large-identifier fixture, not an observation from a real application. Each UTF-16 database is 26,918,912 bytes, below the 256 MiB input allowance; each schema record and statement is also below the native 16 MiB limits. No source corruption was observed.

Selecting one known table with the default selected_table scope succeeded in each encoding and returned its complete single-column schema with zero rows. That is a demonstrated workaround for inspecting a known table; it does not return a whole-database inventory in one call. Real-world incidence is unknown.

REA version

Source revision 2443b949a48c3b2eba687e7381682e49e909d5b9 (package.json: 6.7.0). Only this clone's source was used.

Runtime and platform

Node.js v24.22.1, bundled SQLite 3.53.4, Linux x64. Repository-locked dependencies; no installed REA executable was used.

Target and artifact context

Caller-created, offline SQLite fixtures: UTF-8, UTF-16le, and UTF-16be; 34 identically named ordinary empty tables. Total table-name characters: 4,456,540. UTF-8 fixture: 13,533,184 bytes; each UTF-16 fixture: 26,918,912 bytes.

Interface

Other — direct production inspection boundary and actual SQLite worker JSON protocol. CLI/MCP adapters were not exercised.

Earliest failure stage

Capture or observation

Relevant output or evidence

Captured production-boundary results:

UTF-8 inventory: tables=34, worker_reply_json_bytes=4463359
UTF-8 database: tables=34, worker_reply_json_bytes=8925066
UTF-16le integrity_check: ok
UTF-16be integrity_check: ok
UTF-16le inventory: output-limit
UTF-16le database: output-limit
UTF-16be inventory: output-limit
UTF-16be database: output-limit

Actual source-worker reply for UTF-16le, with exit status 0 and empty stderr:

{"ok":false,"reason":"output-limit","message":"Complete SQLite evidence exceeds the 16 MiB JSON reply budget; no partial success was returned."}

Measured on the successful UTF-8 reports, changing only the database_encoding field to either UTF-16 label gives 4,463,362 bytes for inventory and 8,925,069 bytes for database scope. These are calculated equivalent reply sizes, not successful UTF-16 replies.

Analysis or suspected cause

Source-derived cause: sqliteNameBytesKey converts each encoded name to hex. readSchema charges those internal hex strings to a separate OutputBudget whenever the whole catalog is retained. For ASCII identifiers, a UTF-16 key uses four JSON bytes per name character, while the emitted name uses one. The internal keys exceed the allowance before the returned schema is built. OutputBudget.retain consequently reports a complete-reply limit for data that is not part of the reply.

No source changes or bypassed checks were used.

Additional context

Closed #4184 concerns unrelated catalog names exhausting a selected-table request. The selected-table control succeeds here; the remaining failure concerns whole-database inventory and detailed-schema scopes charging internal hex keys, with a database-encoding-dependent refusal. Open #6855 and #6604 concern native bind-variable and application-collation failures, respectively. No matching report was found in the preliminary searches.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A directly observed valid-schema compatibility failure refuses complete inventory and detailed-schema results well below the documented JSON allowance. The demonstrated input has unusually long identifiers; real-application frequency is unknown.
Workaround status: unverified
Workaround basis: A selected-table request using one known exact table name succeeded on the same UTF-16 fixtures and preserved that table's complete schema. No workaround preserving whole-database inventory was verified; users must already know the relevant table names for the demonstrated alternative.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6888 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspectSqliteDatabaseSnapshot` rejects valid UTF-16le and UTF-16be databases as exceeding the 16 MiB JSON reply budget, although the equivalent UTF-8 schema produces a 4,463,359-byte inventory worker reply or an 8,925,066-byte detailed-schema worker reply. Both UTF-16 fixtures passed SQLite `PRAGMA integrity_check`. The actual source worker also returned `ok: false, reason: "output-limit"` for the UTF-16le inventory. The fixture contains 34 ordinary tables with long ASCII names and one `TEXT` column each. There are no rows, application functions, views, extensions, or custom collations. ## Reproduction From a checkout named `rea`, save this probe beside the checkout as `sqlite-inventory-budget-probe.mjs`. This is the fixture and production-boundary invocation used in the captured run (output reduced here): ```js import { DatabaseSync } from 'node:sqlite'; import { rmSync } from 'node:fs'; import { inspectSqliteDatabaseSnapshot } from './rea/src/sqlite/SqliteDatabaseInspection.ts'; import { jsonByteLength } from './rea/src/domain/jsonSerialization.ts'; const names = Array.from({ length: 34 }, (_, i) => `${i}_${'x'.repeat(128 * 1024)}`); for (const encoding of ['UTF-8', 'UTF-16le', 'UTF-16be']) { const path = new URL(`./sqlite-inventory-budget-${encoding}.db`, import.meta.url).pathname; rmSync(path, { force: true }); const db = new DatabaseSync(path); db.exec(`PRAGMA encoding='${encoding}'; BEGIN;`); for (const name of names) db.exec(`CREATE TABLE "${name}"(value TEXT)`); db.exec('COMMIT'); console.log(encoding, db.prepare('PRAGMA integrity_check').all()); db.close(); for (const schema_scope of ['inventory', 'database']) { try { const inspection = inspectSqliteDatabaseSnapshot(path, { path, schema_scope }); console.log(encoding, schema_scope, inspection.schema.tables.length, jsonByteLength({ ok: true, inspection })); } catch (error) { console.log(encoding, schema_scope, error.reason, error.message); } } } ``` Command used, expressed relative to the clone root: ```sh mise exec node@24.22.1 -- node --import ./node_modules/tsx/dist/loader.mjs ../sqlite-inventory-budget-probe.mjs ``` Observed once per encoding/scope. UTF-8 succeeded in both scopes; both UTF-16 encodings failed in both scopes. A separate invocation of the clone's `src/sqlite/SqliteDatabaseWorker.ts` reproduced the UTF-16le inventory refusal in its JSON protocol. ## Expected behavior Apply the JSON reply allowance to the encoded returned facts. Internal metadata keys should not impose a smaller, encoding-dependent reply limit. Preserve the supported database encoding and complete schema rather than reporting an oversized reply for this fixture. The [documented compact inventory and detailed-schema scopes](https://github.com/morluto/rea/blob/2443b949a48c3b2eba687e7381682e49e909d5b9/docs/sqlite-databases.md#returned-evidence) are both below the documented 16 MiB worker reply allowance here. The existing [large-name schema regression](https://github.com/morluto/rea/blob/2443b949a48c3b2eba687e7381682e49e909d5b9/tests/boundary/sqlite/sqliteDatabaseInspection.test.ts#L223-L242) also expects schemas with many long names to succeed when their complete JSON fits. ## Impact and scope Bounded compatibility failure for schemas with enough retained name bytes. This is an artificial large-identifier fixture, not an observation from a real application. Each UTF-16 database is 26,918,912 bytes, below the 256 MiB input allowance; each schema record and statement is also below the native 16 MiB limits. No source corruption was observed. Selecting one known table with the default `selected_table` scope succeeded in each encoding and returned its complete single-column schema with zero rows. That is a demonstrated workaround for inspecting a known table; it does not return a whole-database inventory in one call. Real-world incidence is unknown. ## REA version Source revision `2443b949a48c3b2eba687e7381682e49e909d5b9` (`package.json`: 6.7.0). Only this clone's source was used. ## Runtime and platform Node.js v24.22.1, bundled SQLite 3.53.4, Linux x64. Repository-locked dependencies; no installed REA executable was used. ## Target and artifact context Caller-created, offline SQLite fixtures: UTF-8, UTF-16le, and UTF-16be; 34 identically named ordinary empty tables. Total table-name characters: 4,456,540. UTF-8 fixture: 13,533,184 bytes; each UTF-16 fixture: 26,918,912 bytes. ## Interface Other — direct production inspection boundary and actual SQLite worker JSON protocol. CLI/MCP adapters were not exercised. ## Earliest failure stage Capture or observation ## Relevant output or evidence Captured production-boundary results: ```text UTF-8 inventory: tables=34, worker_reply_json_bytes=4463359 UTF-8 database: tables=34, worker_reply_json_bytes=8925066 UTF-16le integrity_check: ok UTF-16be integrity_check: ok UTF-16le inventory: output-limit UTF-16le database: output-limit UTF-16be inventory: output-limit UTF-16be database: output-limit ``` Actual source-worker reply for UTF-16le, with exit status 0 and empty stderr: ```json {"ok":false,"reason":"output-limit","message":"Complete SQLite evidence exceeds the 16 MiB JSON reply budget; no partial success was returned."} ``` Measured on the successful UTF-8 reports, changing only the `database_encoding` field to either UTF-16 label gives 4,463,362 bytes for inventory and 8,925,069 bytes for database scope. These are calculated equivalent reply sizes, not successful UTF-16 replies. ## Analysis or suspected cause Source-derived cause: [`sqliteNameBytesKey`](https://github.com/morluto/rea/blob/2443b949a48c3b2eba687e7381682e49e909d5b9/src/sqlite/SqliteDatabaseInspection.ts#L125-L128) converts each encoded name to hex. [`readSchema`](https://github.com/morluto/rea/blob/2443b949a48c3b2eba687e7381682e49e909d5b9/src/sqlite/SqliteDatabaseInspection.ts#L315-L329) charges those internal hex strings to a separate `OutputBudget` whenever the whole catalog is retained. For ASCII identifiers, a UTF-16 key uses four JSON bytes per name character, while the emitted name uses one. The internal keys exceed the allowance before the returned schema is built. [`OutputBudget.retain`](https://github.com/morluto/rea/blob/2443b949a48c3b2eba687e7381682e49e909d5b9/src/sqlite/SqliteDatabaseInspection.ts#L78-L90) consequently reports a complete-reply limit for data that is not part of the reply. No source changes or bypassed checks were used. ## Additional context Closed [#4184](https://github.com/morluto/rea/issues/4184) concerns unrelated catalog names exhausting a selected-table request. The selected-table control succeeds here; the remaining failure concerns whole-database inventory and detailed-schema scopes charging internal hex keys, with a database-encoding-dependent refusal. Open #6855 and #6604 concern native bind-variable and application-collation failures, respectively. No matching report was found in the preliminary searches. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A directly observed valid-schema compatibility failure refuses complete inventory and detailed-schema results well below the documented JSON allowance. The demonstrated input has unusually long identifiers; real-application frequency is unknown. - Workaround status: unverified - Workaround basis: A selected-table request using one known exact table name succeeded on the same UTF-16 fixtures and preserved that table's complete schema. No workaround preserving whole-database inventory was verified; users must already know the relevant table names for the demonstrated alternative.

bsakaguchi closed this issue

2026-10-31 00:47:50 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:07 +00:00

[PR #4222] [CLOSED] fix(website): translate coding agent as an AI assistant in Russian and Ukrainian #6134

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425