[PR #2489] [MERGED] test(perf): verify existing SQLite column selection on QGIS data #5267

New issue

Closed

opened 2026-10-31 01:05:18 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:05:18 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2489
Author: @N0zoM1z0
Created: 10/13/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/2016-real-data-round-20261013

📝 Commits (4)

93dbb27 test(perf): measure existing SQLite column selection on QGIS data
bfabc70 fix(experiments): approve the selected static SQLite trial tool
44c7e88 docs(perf): distinguish frozen measurements from current-main checks
8b5c87e fix(experiments): preserve SQLite fact presence and JSON types

📊 Changes

7 files changed (+264 additions, -1 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+4 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+1 -0)
📝 experiments/analysis-efficiency/corpus.json (+13 -0)
📝 experiments/analysis-efficiency/measure.mjs (+58 -0)
➕ experiments/analysis-efficiency/questions/sqlite-qgis.txt (+1 -0)
📝 experiments/analysis-efficiency/review-model.py (+6 -1)
➕ experiments/analysis-efficiency/sqlite-existing-controls.md (+181 -0)

📄 Description

Measure existing SQLite column selection on QGIS's shipped world-map GeoPackage before adding another efficiency API. The same main runtime preserves complete country-table schema and exact tagged sample facts while existing columns reduce the SDK reply 96.2%. No production, API/default, annotation, skill guidance or cache change is justified or included.

Two actual Codex pairs pass selected factual checks: total input falls 57.63% / 36.44%, uncached 54.14% / 44.08%, but time is mixed (4.17% slower / 14.39% faster). SDK time/CPU do not improve. Keep all repeated calls and four initial approval-blocked trials; the experiment-only driver adds its existing per-tool approval override for SQLite with one exact tool enabled. Global configuration and REA authority stay intact.

Add pinned corpus, a reproducible existing-selection workflow, fixed question, factual grader and curated results. Full original results stay private, with no downloaded binary or raw results committed. Real CLI/MCP inventory/all/columns parity and an independent read-only SQLite binding confirm SQL/columns/sample. Six context/driver regressions, JS syntax and Python compilation pass; bounded full-project pre-push typecheck/lint, docs checks and a real post-format SDK smoke pass. Rebased latest-main build, real SDK all/columns fact comparison, source-byte preservation and docs checks also pass. Missing final resource counters remain unknown.

Refs #2016. No release/deployment.

Measured runtime is frozen at 172407787f1842dedbcd27b5385177595f786ce4; the branch is rebased onto bb540826d65316c164d2de8ba4ef10f81a8ad073. The five new commits concern archive contracts, schema testing, pwntools cleanup, Mach-O concurrency and DMG retries; none changes SQLite selection. Integration checks do not relabel earlier measurements.

Manual review tightens the SQLite factual grader to distinguish omitted fields from explicit null and preserve JSON scalar types. Regrading all four stored model answers passes; deliberate missing-wal and boolean-row-limit mutations fail. Six context/driver regressions pass again. Original trials/reviews remain retained and no model reruns are used. Automated Codex review reports exhausted review quota, so it did not execute; manual diff/transcript review and CI remain explicit.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2489 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/13/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/2016-real-data-round-20261013` --- ### 📝 Commits (4) - [`93dbb27`](https://github.com/morluto/rea/commit/93dbb2734524585ecd0d6ce9c42217ad9737b374) test(perf): measure existing SQLite column selection on QGIS data - [`bfabc70`](https://github.com/morluto/rea/commit/bfabc7054ad805d5ba8be720327ab21577ce52bd) fix(experiments): approve the selected static SQLite trial tool - [`44c7e88`](https://github.com/morluto/rea/commit/44c7e88fb83e80ebc43817042bf23de1a2aa13d0) docs(perf): distinguish frozen measurements from current-main checks - [`8b5c87e`](https://github.com/morluto/rea/commit/8b5c87e80bd4b75f9638d0424ef81e492b97b962) fix(experiments): preserve SQLite fact presence and JSON types ### 📊 Changes **7 files changed** (+264 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+4 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+1 -0) 📝 `experiments/analysis-efficiency/corpus.json` (+13 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+58 -0) ➕ `experiments/analysis-efficiency/questions/sqlite-qgis.txt` (+1 -0) 📝 `experiments/analysis-efficiency/review-model.py` (+6 -1) ➕ `experiments/analysis-efficiency/sqlite-existing-controls.md` (+181 -0) </details> ### 📄 Description Measure existing SQLite column selection on QGIS's shipped world-map GeoPackage before adding another efficiency API. The same main runtime preserves complete country-table schema and exact tagged sample facts while existing columns reduce the SDK reply 96.2%. No production, API/default, annotation, skill guidance or cache change is justified or included. Two actual Codex pairs pass selected factual checks: total input falls 57.63% / 36.44%, uncached 54.14% / 44.08%, but time is mixed (4.17% slower / 14.39% faster). SDK time/CPU do not improve. Keep all repeated calls and four initial approval-blocked trials; the experiment-only driver adds its existing per-tool approval override for SQLite with one exact tool enabled. Global configuration and REA authority stay intact. Add pinned corpus, a reproducible existing-selection workflow, fixed question, factual grader and curated results. Full original results stay private, with no downloaded binary or raw results committed. Real CLI/MCP inventory/all/columns parity and an independent read-only SQLite binding confirm SQL/columns/sample. Six context/driver regressions, JS syntax and Python compilation pass; bounded full-project pre-push typecheck/lint, docs checks and a real post-format SDK smoke pass. Rebased latest-main build, real SDK all/columns fact comparison, source-byte preservation and docs checks also pass. Missing final resource counters remain unknown. Refs #2016. No release/deployment. Measured runtime is frozen at `172407787f1842dedbcd27b5385177595f786ce4`; the branch is rebased onto `bb540826d65316c164d2de8ba4ef10f81a8ad073`. The five new commits concern archive contracts, schema testing, pwntools cleanup, Mach-O concurrency and DMG retries; none changes SQLite selection. Integration checks do not relabel earlier measurements. Manual review tightens the SQLite factual grader to distinguish omitted fields from explicit null and preserve JSON scalar types. Regrading all four stored model answers passes; deliberate missing-wal and boolean-row-limit mutations fail. Six context/driver regressions pass again. Original trials/reviews remain retained and no model reruns are used. Automated Codex review reports exhausted review quota, so it did not execute; manual diff/transcript review and CI remain explicit. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:05:18 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5267

No description provided.

Rows
Columns