morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4127] [Bug] inspect_sqlite_database reports the 256 MiB snapshot budget as invalid_request instead of resource_constraint #1624

New issue

Closed

opened 2026-10-16 21:08:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-16 21:08:08 +00:00

Owner

Originally created by @Rinkia on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4127

Before submitting

I searched the open and closed issues for a related report (sqlite snapshot budget, sqlite resource_constraint, out_of_range resource_constraint): none found.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A database larger than the documented 256 MiB database+WAL budget is refused as a generic input error:

{
  "code": "invalid_request",
  "category": "invalid_input",
  "message": "Analysis input is invalid. Check the arguments and try again.",
  "retryable": true,
  "remediation": { "action": "Correct the listed arguments and retry." },
  "details": {
    "operation": "inspect_sqlite_database",
    "issues": [{ "path": ["path"], "reason": "out_of_range",
      "message": "Selected SQLite database and WAL exceed the combined 268435456-byte snapshot budget: …/huge.db" }]
  }
}

The same tool reports its other byte limits as resource constraints. A 20 MiB blob gives:

{ "code": "resource_constraint", "category": "resource_constraint", "retryable": false,
  "details": { "resource": "transport",
    "reason": "Selected row byte lengths exceed the 16 MiB reply budget before hex projection.",
    "reported_limits": { "sqlite_value_or_record_bytes": 16777216, "worker_reply_bytes": 16777216 } } }

Other providers also report their input snapshot budgets as resource_constraint with structured limits:

Flutter: "The selected APK exceeds the …-byte snapshot budget", { max_apk_bytes, observed_bytes }.
Apple dispatch metadata: "…exceeds its 64 MiB snapshot budget".
JavaScript workflow input: resource_constraint with input_reason: "too-large".
Managed code: "Input overflow returns a resource constraint" (docs/managed-code-analysis.md).

So for the SQLite input budget:

retryable: true and "Correct the listed arguments" suggest the call might succeed if repeated or reworded. It can't; the file has to get smaller.
A client branching on code/category to detect resource limits (e.g. to fall back to a smaller snapshot) misses this case.
The limit and observed size exist only inside the message text, not as reported_limits.

Cause

src/sqlite/SqliteDatabaseSnapshot.ts (~line 209, main @ fe7bcc3fb) throws ArtifactReaderFailure("limit", …) for the combined size check. src/sqlite/SqliteDatabaseFailures.ts (~lines 79-85) then maps every non-cleanup ArtifactReaderFailure to AnalysisInputError, turning "limit" into reason: "out_of_range". This mapping dates from #2271.

Reproduction

# python3 make_huge.py -> huge.db (~258 MiB of zeroblobs; any database over 256 MiB works)
import sqlite3
con = sqlite3.connect("huge.db")
con.execute("create table big(v)")
for _ in range(270):
    con.execute("insert into big values (zeroblob(1000000))")
con.commit()

rea inspect-sqlite-database ./huge.db --json   # invalid_request / invalid_input / retryable true (above)

Expected behavior

Exceeding the database+WAL snapshot budget returns resource_constraint, retryable: false, with reported_limits (e.g. { snapshot_input_bytes: 268435456 }), the observed byte count, and a remediation such as "supply a smaller snapshot (e.g. export the relevant tables with the application's backup/export path)".
Other ArtifactReaderFailure reasons (format, identity) stay as input errors.

Impact and scope

Affects any database or database+WAL pair over 256 MiB, which is common for real application stores (browser history, messaging apps, mobile backups).
Contract consistency only: no wrong data is returned, and the message text names the limit.
Deterministic.

REA version

Executed on rea-agents@6.5.0. The mapping is unchanged on main @ fe7bcc3fb (source-derived).

Runtime and platform

rea Docker image (Linux x64, Node 24.21.0, SQLite 3.53.4 as reported by REA) on Windows 11 / Docker Desktop.

Target and artifact context

Synthetic database from the script above, 270,467,072 bytes.

Interface

CLI (MCP shares the provider).

Relevant output or evidence

For context, everything else I checked in this tool matched Python sqlite3 (typeof/hex) byte for byte:

Data: the Chinook sample (8,715 rows); 64-bit integer extremes; -0 and ±Infinity; invalid UTF-8 and lone surrogates in UTF-8, UTF-16LE and UTF-16BE databases; empty and quoted identifiers.
Table kinds: generated, WITHOUT ROWID and STRICT tables; FTS5/R-tree virtual and shadow classification; an unknown virtual-table module.
Journals and size: WAL-only committed rows; a garbage WAL tail ignored; a hot rollback journal refused; a PERSIST zeroed journal accepted; 64 KiB pages; 5,000 tables.

Originally created by @Rinkia on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4127 ### Before submitting - [x] I searched the open and closed issues for a related report (sqlite snapshot budget, sqlite resource_constraint, out_of_range resource_constraint): none found. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A database larger than the documented 256 MiB database+WAL budget is refused as a generic input error: ```json { "code": "invalid_request", "category": "invalid_input", "message": "Analysis input is invalid. Check the arguments and try again.", "retryable": true, "remediation": { "action": "Correct the listed arguments and retry." }, "details": { "operation": "inspect_sqlite_database", "issues": [{ "path": ["path"], "reason": "out_of_range", "message": "Selected SQLite database and WAL exceed the combined 268435456-byte snapshot budget: …/huge.db" }] } } ``` The same tool reports its other byte limits as resource constraints. A 20 MiB blob gives: ```json { "code": "resource_constraint", "category": "resource_constraint", "retryable": false, "details": { "resource": "transport", "reason": "Selected row byte lengths exceed the 16 MiB reply budget before hex projection.", "reported_limits": { "sqlite_value_or_record_bytes": 16777216, "worker_reply_bytes": 16777216 } } } ``` Other providers also report their input snapshot budgets as `resource_constraint` with structured limits: - Flutter: "The selected APK exceeds the …-byte snapshot budget", `{ max_apk_bytes, observed_bytes }`. - Apple dispatch metadata: "…exceeds its 64 MiB snapshot budget". - JavaScript workflow input: `resource_constraint` with `input_reason: "too-large"`. - Managed code: "Input overflow returns a resource constraint" (`docs/managed-code-analysis.md`). So for the SQLite input budget: - `retryable: true` and "Correct the listed arguments" suggest the call might succeed if repeated or reworded. It can't; the file has to get smaller. - A client branching on `code`/`category` to detect resource limits (e.g. to fall back to a smaller snapshot) misses this case. - The limit and observed size exist only inside the message text, not as `reported_limits`. ### Cause `src/sqlite/SqliteDatabaseSnapshot.ts` (~line 209, `main` @ `fe7bcc3fb`) throws `ArtifactReaderFailure("limit", …)` for the combined size check. `src/sqlite/SqliteDatabaseFailures.ts` (~lines 79-85) then maps every non-cleanup `ArtifactReaderFailure` to `AnalysisInputError`, turning `"limit"` into `reason: "out_of_range"`. This mapping dates from #2271. ### Reproduction ```python # python3 make_huge.py -> huge.db (~258 MiB of zeroblobs; any database over 256 MiB works) import sqlite3 con = sqlite3.connect("huge.db") con.execute("create table big(v)") for _ in range(270): con.execute("insert into big values (zeroblob(1000000))") con.commit() ``` ```sh rea inspect-sqlite-database ./huge.db --json # invalid_request / invalid_input / retryable true (above) ``` ### Expected behavior - Exceeding the database+WAL snapshot budget returns `resource_constraint`, `retryable: false`, with `reported_limits` (e.g. `{ snapshot_input_bytes: 268435456 }`), the observed byte count, and a remediation such as "supply a smaller snapshot (e.g. export the relevant tables with the application's backup/export path)". - Other `ArtifactReaderFailure` reasons (format, identity) stay as input errors. ### Impact and scope - Affects any database or database+WAL pair over 256 MiB, which is common for real application stores (browser history, messaging apps, mobile backups). - Contract consistency only: no wrong data is returned, and the message text names the limit. - Deterministic. ### REA version Executed on `rea-agents@6.5.0`. The mapping is unchanged on `main` @ `fe7bcc3fb` (source-derived). ### Runtime and platform `rea` Docker image (Linux x64, Node 24.21.0, SQLite 3.53.4 as reported by REA) on Windows 11 / Docker Desktop. ### Target and artifact context Synthetic database from the script above, 270,467,072 bytes. ### Interface CLI (MCP shares the provider). ### Relevant output or evidence For context, everything else I checked in this tool matched Python `sqlite3` (`typeof`/`hex`) byte for byte: - Data: the Chinook sample (8,715 rows); 64-bit integer extremes; `-0` and ±Infinity; invalid UTF-8 and lone surrogates in UTF-8, UTF-16LE and UTF-16BE databases; empty and quoted identifiers. - Table kinds: generated, WITHOUT ROWID and STRICT tables; FTS5/R-tree virtual and shadow classification; an unknown virtual-table module. - Journals and size: WAL-only committed rows; a garbage WAL tail ignored; a hot rollback journal refused; a PERSIST zeroed journal accepted; 64 KiB pages; 5,000 tables.

bsakaguchi closed this issue

2026-10-16 21:08:08 +00:00

bsakaguchi commented

2026-10-16 21:08:10 +00:00

Author

Owner

@coder6955 commented on GitHub (Oct 15, 2026):

I'd like to work on this. I'll reproduce the oversized database and database-plus-WAL cases, preserve the observed size at the snapshot boundary, and check that format and identity failures keep their current classifications.

@coder6955 commented on GitHub (Oct 15, 2026): I'd like to work on this. I'll reproduce the oversized database and database-plus-WAL cases, preserve the observed size at the snapshot boundary, and check that format and identity failures keep their current classifications.

bsakaguchi commented

2026-10-16 21:08:10 +00:00

Author

Owner

@coder6955 commented on GitHub (Oct 15, 2026):

I saw #4141 opened while I was finishing a producer-boundary version, so I won't open a duplicate PR. The implementation and regression tests are available in fc79b67 if useful for that work.

This version raises the typed resource error where the combined size is measured, retaining the exact observed byte count and a smaller-snapshot remediation. It covers an oversized database, oversized WAL, combined size, and matching CLI/MCP errors.

Linux validation on the pinned Node 24.18.0/npm 11.16.0 toolchain:

The three added snapshot regressions failed on the original producer; its 16 existing cases passed.
npm run test:sqlite: 122 passed, 3 conditional skips, no failures. Both new CLI/MCP cases passed.
npm run check and npm run docs:check passed.
A real SQLite database containing 270 zeroblob(1000000) rows measured 270,405,632 bytes. The compiled CLI returned resource_constraint, retryable: false, the 268,435,456-byte limit, and the exact observed size.

Logs and the CLI result are attached to that run as sqlite-4127-linux-evidence. Feel free to reuse the regression coverage or compare the producer-side handling with #4141's mapper-side approach.

@coder6955 commented on GitHub (Oct 15, 2026): I saw #4141 opened while I was finishing a producer-boundary version, so I won't open a duplicate PR. The implementation and regression tests are available in [fc79b67](https://github.com/coder6955/rea/commit/fc79b673803e828c4bd47cf012b2e726abd8eecd) if useful for that work. This version raises the typed resource error where the combined size is measured, retaining the exact observed byte count and a smaller-snapshot remediation. It covers an oversized database, oversized WAL, combined size, and matching CLI/MCP errors. [Linux validation](https://github.com/coder6955/rea/actions/runs/38750057571) on the pinned Node 24.18.0/npm 11.16.0 toolchain: - The three added snapshot regressions failed on the original producer; its 16 existing cases passed. - `npm run test:sqlite`: 122 passed, 3 conditional skips, no failures. Both new CLI/MCP cases passed. - `npm run check` and `npm run docs:check` passed. - A real SQLite database containing 270 `zeroblob(1000000)` rows measured 270,405,632 bytes. The compiled CLI returned `resource_constraint`, `retryable: false`, the 268,435,456-byte limit, and the exact observed size. Logs and the CLI result are attached to that run as `sqlite-4127-linux-evidence`. Feel free to reuse the regression coverage or compare the producer-side handling with #4141's mapper-side approach.

bsakaguchi commented

2026-10-16 21:08:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 15, 2026):

Fixed on main in bc51d0ea5. The combined database/WAL byte budget is now measured and typed at the snapshot owner before file acquisition or worker execution. It reports resource_constraint, retryable=false, the 268435456-byte budget, exact observed bytes, and supported smaller-snapshot export guidance.

A sparse database-plus-WAL case passed through both CLI and stdio MCP; a separate producer case preserves exact sizes above JavaScript's safe-integer range. Late growth remains changed evidence, with cleanup and partial facts retained. Closing the reported classification defect.

Implementation qualification passed: CI at d9aa06d93. This verifies the implementation revision; the later PCAP-only batch has its own checks.

@morluto commented on GitHub (Oct 15, 2026): Fixed on main in bc51d0ea5. The combined database/WAL byte budget is now measured and typed at the snapshot owner before file acquisition or worker execution. It reports resource_constraint, retryable=false, the 268435456-byte budget, exact observed bytes, and supported smaller-snapshot export guidance. A sparse database-plus-WAL case passed through both CLI and stdio MCP; a separate producer case preserves exact sizes above JavaScript's safe-integer range. Late growth remains changed evidence, with cleanup and partial facts retained. Closing the reported classification defect. Implementation qualification passed: [CI at d9aa06d93](https://github.com/morluto/rea/actions/runs/38797211865). This verifies the implementation revision; the later PCAP-only batch has its own checks.

bsakaguchi referenced this issue

2026-10-31 00:54:13 +00:00

[PR #1624] [MERGED] test(clients): verify Copilot compact schema workflows #4709

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1624

No description provided.

Rows
Columns