morluto/rea
1
0
Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00
Code Issues 1.3k Projects Packages Wiki Activity

[PR #2851] [CLOSED] fix(verify): route OpenAI model fixture requests by pathname #5437

New issue
Closed
opened 2026-10-31 01:06:37 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-31 01:06:37 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2851
Author: @shubhamahuja9999
Created: 10/13/2026
Status: ❌ Closed

Base: main ← Head: fix/model-fixture-query-routes

📝 Commits (2)

  • 154e1c2 fix(verify): route OpenAI model fixture requests by pathname
  • 2be0921 fix(verify): type the OpenAI fixture's assistant message callback

📊 Changes

3 files changed (+100 additions, -2 deletions)

View changed files

➕ scripts/verify/clients/model-fixture.d.mts (+44 -0)
📝 scripts/verify/clients/model-fixture.mjs (+4 -2)
➕ tests/evaluation/model-evals/openAiModelFixture.test.ts (+52 -0)

📄 Description

Summary

Fixes #2588

The loopback OpenAI model fixture (scripts/verify/clients/model-fixture.mjs) used by the native client verification lanes compared the raw request target, query included. A chat-completion request such as POST /v1/chat/completions?api-version=1 was treated as an unsupported endpoint and returned 404 without reaching onRequest. GET /v1/models?limit=1 had the same problem.

Problem and expected behavior

Smallest trigger: POST {baseUrl}/chat/completions?stream=true returns 404, while the same body without the query returns 200.

Expected: the query component does not select an endpoint. Both chat-completion requests reach the callback and return 200, model listing ignores the query, and unrelated endpoints such as /v1/completions still return 404. This follows the format-aware boundary parsing rule in AGENTS.md and matches the sibling anthropic-model-fixture.mjs and gemini-model-fixture.mjs, which already route by new URL(request.url, "http://localhost").pathname.

Change and scope

  • Parse the pathname once per request and use it for both the /models and /v1/chat/completions routes.
  • Recorded requests[].path, probes[].path and the incoming-request files keep the original request target, query included, so captured client behavior is preserved.
  • Add the fixture's first regression test and a model-fixture.d.mts declaration so TypeScript tests can import it, following scripts/lib/*.d.mts. The declaration types the onRequest assistant message (role, content, optional tool_calls), so a callback returning nothing fails type checking (review follow-up).
  • Not included: streaming still follows the request body, and no other fixture behavior changes.

Placement note: no test covered these client fixtures before, so the test sits with the other model-evaluation tests in tests/evaluation/model-evals/. Happy to move it if you prefer another home.

Contribution scope: verification-fixture fix within existing boundaries; maintainer decision not applicable.

Contract and boundary impact

  • Semantic owner and earliest changed stage: request routing in the loopback OpenAI fixture.
  • Contribution scope fit and linked maintainer decision: not applicable.
  • CLI and MCP/tool-catalog contract: none.
  • Provider, bridge, target-format, or platform compatibility: none; affects only verification lanes that use this fixture.
  • Evidence, artifact, provenance, or reconstruction contract: none.
  • Target selection, authority, upstream behavior, and cleanup details: the fixture still binds 127.0.0.1 on an ephemeral port; the test closes it in finally and uses createTestTempDirectory.
  • Generated metadata, package, or installation impact: none.

Expected effects:

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

None of these apply: this is a test fixture behind loopback verification lanes.

Evidence and regression coverage

  • Test added: tests/evaluation/model-evals/openAiModelFixture.test.ts sends /chat/completions, /chat/completions?api-version=1, /completions and GET /models?limit=1 to the real fixture. It asserts statuses [200, 200, 404, 200], two callback invocations, the recorded original paths, one 404 probe for /v1/completions, and no fixture failure.
  • Base reproduction: without the fix the test fails with statuses [200, 404, 404, 404], which reproduces the issue and the same /models query case.
  • User-visible output: not applicable.
  • Remaining proof gaps: no native client lane (verify:*-client) was run; those need installed third-party clients.

Validation performed

Windows 11, Node.js 24.18.0, npm 11.16.0, on main at 6e25cc3c7:

  • npm run test:local -- tests/evaluation/model-evals/openAiModelFixture.test.ts: failed without the fix ([200, 404, 404, 404]); passes with it.
  • npm run test:focused -- tests/evaluation/model-evals/openAiModelFixture.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts: 23/23 passed.
  • npm run test:local -- all 4 tests/evaluation/model-evals/*.test.ts: 71/71 passed.
  • npm run check:fast (typecheck and lint): passed.
  • npm run verify:test-discovery: "Discovered all 1089 test files exactly once."
  • oxfmt --check on changed files: passed.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable.
  • Package or release metadata impact: none; scripts/verify/ is not shipped.
  • Security, privacy, process, or containment review: loopback-only fixture; recorded paths unchanged.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2851 **Author:** [@shubhamahuja9999](https://github.com/shubhamahuja9999) **Created:** 10/13/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/model-fixture-query-routes` --- ### 📝 Commits (2) - [`154e1c2`](https://github.com/morluto/rea/commit/154e1c216634687565f7800e428c17c6582abfec) fix(verify): route OpenAI model fixture requests by pathname - [`2be0921`](https://github.com/morluto/rea/commit/2be0921a8de125eeb468b23f5d82723280a51c78) fix(verify): type the OpenAI fixture's assistant message callback ### 📊 Changes **3 files changed** (+100 additions, -2 deletions) <details> <summary>View changed files</summary> ➕ `scripts/verify/clients/model-fixture.d.mts` (+44 -0) 📝 `scripts/verify/clients/model-fixture.mjs` (+4 -2) ➕ `tests/evaluation/model-evals/openAiModelFixture.test.ts` (+52 -0) </details> ### 📄 Description ## Summary Fixes #2588 The loopback OpenAI model fixture (`scripts/verify/clients/model-fixture.mjs`) used by the native client verification lanes compared the raw request target, query included. A chat-completion request such as `POST /v1/chat/completions?api-version=1` was treated as an unsupported endpoint and returned 404 without reaching `onRequest`. `GET /v1/models?limit=1` had the same problem. ## Problem and expected behavior Smallest trigger: `POST {baseUrl}/chat/completions?stream=true` returns 404, while the same body without the query returns 200. Expected: the query component does not select an endpoint. Both chat-completion requests reach the callback and return 200, model listing ignores the query, and unrelated endpoints such as `/v1/completions` still return 404. This follows the format-aware boundary parsing rule in AGENTS.md and matches the sibling `anthropic-model-fixture.mjs` and `gemini-model-fixture.mjs`, which already route by `new URL(request.url, "http://localhost").pathname`. ## Change and scope - Parse the pathname once per request and use it for both the `/models` and `/v1/chat/completions` routes. - Recorded `requests[].path`, `probes[].path` and the incoming-request files keep the original request target, query included, so captured client behavior is preserved. - Add the fixture's first regression test and a `model-fixture.d.mts` declaration so TypeScript tests can import it, following `scripts/lib/*.d.mts`. The declaration types the `onRequest` assistant message (`role`, `content`, optional `tool_calls`), so a callback returning nothing fails type checking (review follow-up). - Not included: streaming still follows the request body, and no other fixture behavior changes. Placement note: no test covered these client fixtures before, so the test sits with the other model-evaluation tests in `tests/evaluation/model-evals/`. Happy to move it if you prefer another home. Contribution scope: verification-fixture fix within existing boundaries; maintainer decision not applicable. ## Contract and boundary impact - Semantic owner and earliest changed stage: request routing in the loopback OpenAI fixture. - Contribution scope fit and linked maintainer decision: not applicable. - CLI and MCP/tool-catalog contract: none. - Provider, bridge, target-format, or platform compatibility: none; affects only verification lanes that use this fixture. - Evidence, artifact, provenance, or reconstruction contract: none. - Target selection, authority, upstream behavior, and cleanup details: the fixture still binds `127.0.0.1` on an ephemeral port; the test closes it in `finally` and uses `createTestTempDirectory`. - Generated metadata, package, or installation impact: none. Expected effects: - [ ] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. None of these apply: this is a test fixture behind loopback verification lanes. ## Evidence and regression coverage - Test added: `tests/evaluation/model-evals/openAiModelFixture.test.ts` sends `/chat/completions`, `/chat/completions?api-version=1`, `/completions` and `GET /models?limit=1` to the real fixture. It asserts statuses `[200, 200, 404, 200]`, two callback invocations, the recorded original paths, one 404 probe for `/v1/completions`, and no fixture failure. - Base reproduction: without the fix the test fails with statuses `[200, 404, 404, 404]`, which reproduces the issue and the same `/models` query case. - User-visible output: not applicable. - Remaining proof gaps: no native client lane (`verify:*-client`) was run; those need installed third-party clients. ## Validation performed Windows 11, Node.js 24.18.0, npm 11.16.0, on `main` at `6e25cc3c7`: - `npm run test:local -- tests/evaluation/model-evals/openAiModelFixture.test.ts`: failed without the fix (`[200, 404, 404, 404]`); passes with it. - `npm run test:focused -- tests/evaluation/model-evals/openAiModelFixture.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts`: 23/23 passed. - `npm run test:local --` all 4 `tests/evaluation/model-evals/*.test.ts`: 71/71 passed. - `npm run check:fast` (typecheck and lint): passed. - `npm run verify:test-discovery`: "Discovered all 1089 test files exactly once." - `oxfmt --check` on changed files: passed. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. - Package or release metadata impact: none; `scripts/verify/` is not shipped. - Security, privacy, process, or containment review: loopback-only fixture; recorded paths unchanged. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:06:37 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/procedure-reference-stacked
release-please--branches--main--components--rea-agents
release-please--branches--main--components--rea-agents--release-notes
codex/kuna-independent-retry
codex/byte-search-overlay-coverage
codex/open-failure-target-receipt
codex/kuna-xref-evidence
codex/ghidra-ambiguous-procedure-recovery
codex/function-source-bundle
codex/trace-function-integration-prototype
codex/snapshot-source-closure
codex/recorded-evidence-array-pages
codex/evidence-recovery-redesign
codex/trace-feature-address-scope
codex/call-discovery-scope
codex/ghidra-literal-candidate-evidence
codex/decompile-directory-independent-retry
codex/retry-provenance-base
codex/value-flow-capability-experiment
codex/ghidra-containing-instruction
cursor/readme-discord-button-ef9f
dependabot/github_actions/actions/setup-node-7.1.0
dependabot/github_actions/actions/upload-artifact-7.0.2
dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df
codex/kuna-1989
fix/8559-native-ui-fixture-ownership
fix/8555-go-verifier-mcp-ownership
codex/provider-capability-recovery
fix/8642-rizin-target-stdin
codex/ghidra-memory-effect-scope
codex/ghidra-annotation-entry-selector
codex/ghidra-readonly-recovery
codex/ghidra-regex-verifier-recovery
codex/evidence-discovery-view-guidance
fix/8545-bulk-append-overflow
fix/8428-navigation-refusal-alternatives
codex/address-context-recovery
codex/matching-member-limitations
codex/unknown-scope-contract
codex/evidence-view-guidance
feat/8373-native-trace-views
codex/ghidra-snapshot-verifier-binding
codex/elf-export-visibility
codex/evidence-view-recovery
codex/native-reference-cli-help
codex/provider-candidate-observation-scope
codex/kuna-session-verification
codex/ghidra-switch-lifetime-guidance
codex/pseudocode-paging-discovery
codex/clarify-evidence-view-support
codex/kuna-xrefs-selector-recovery
codex/context-capacity-recovery
codex/evidence-closure-large-links
codex/same-function-evidence-identity
codex/ghidra-verifier-inspected-jdk
codex/audit-guided-workflows
codex/crash-prompt-retained-dossiers
codex/analysis-view-error-recovery
codex/kuna-verifier-target-selection
codex/ghidra-selected-project-call-coverage
codex/docs-native-analysis-scope
codex/ghidra-java-socket-boundary
codex/ghidra-retained-snapshot-verifier
codex/ghidra-search-inventory-facts
fix/8284-zero-argument-parameter-bindings
fix/8198-ui-action-handler-offset
fix/8257-elf-entry-label-identity
fix/8304-deferred-output-schema-projection
fix/8277-call-path-computed-target-uncertainty
fix/pr-8089-conflicts
fix/ci-static-duplicate-export
dependabot/npm_and_yarn/lint-staged-17.6.0
dependabot/github_actions/actions/download-artifact-8.0.2
dependabot/npm_and_yarn/write-file-atomic-8.0.0
dependabot/npm_and_yarn/canonicalize-5.1.0
dependabot/github_actions/actions/cache/restore-6.1.0
dependabot/npm_and_yarn/toon-format/toon-4.4.0
dependabot/github_actions/actions/deploy-pages-5.0.1
codex/review-fixes-pr-2403
codex/review-fixes-pr-2400
codex/process-terminal-width
codex/test-brittleness-cleanup
verify/7167-browser-connect-macos
fix/7167-browser-connect
feat/native-matching-allowance-setting
feat/browser-scenario-dry-run
test/managed-caller-parameters-20261024
codex/parser-fortythird-boundaries
fix/6926-pinvoke-platform-decorations-20261023
fix/6889-har-body-encoding-20261023
fix/6888-sqlite-schema-budget-20261023
fix/dependency-security-gates-6708
cursor/evidence-overwrite-rebase-b4ac
feat/4057-verify-native-exports
fix/windows-acl-observer-overhead-20261022
cursor/erlang-beam-restack-4476
cursor/pe-certificate-review-bf32
fix/6116-javascript-application-fs-errors
fix/6061-unmapped-runtime-script-unknown
feat/browser-scenario-summary
fix/pe-certificates
codex/verify-3407
cursor/rebase-5699
cursor/rebase-5352
cursor/rebase-5600
cursor/fix-cli-schema-omission-10a8
codex/audit-hopper-shutdown-reasons-20261021
fix/demangle-swift-omitted-record
fix/nib-boolean-value-types
cursor/cancellation-partial-observation-a5ac
fix/ib-object-limit-reasons
fix/electron-fuse-limit-advice
fix/kuna-doctor-dangling-cache
fix/fixed-budget-refusal-advice
fix/js-ipc-listener-dedup
fix/js-return-member-provenance
fix/dmg-integrity-retry-advice
fix/ida-doctor-readiness
fix/kuna-version-probe-output-drain
fix/legacy-swift-runtime-name-matching
fix/dylib-lazy-load-commands
fix/browser-scenario-startup-cancel
fix/swift-nested-punycode-classification
fix/compare-unselected-scopes
fix/objc-method-budget-coverage
fix/dmg-mounted-root-observations
audit/ci-tests-20261019-round2
release/6.7.0
fix/wabt-cleanup-retry-resolution
fix/electron-fuse-elf-pe-architecture
fix/native-plist-descriptor-ownership
fix/kuna-caller-cleanup-ownership
experiment/atomic-codemode-20261016
review/first-contributor-3776
fix/call-resolution-full-read-cycles
fix/member-mutation-selection-repeat
experiment/atomic-re-tools-20261015
perf/member-mutation-lazy-origins
perf/static-network-mutation-index
fix/hopper-startup-lifecycle
audit/3050-windows-proof-20261015
feat/windows-window-capture-2183
audit/3049-windows-proof-20261015
audit/3735-macos-fixtures-20261015
fix/batch-decompile-null-pseudocode
fix/kuna-inventory-honors-jobs
experiment/browser-codemode-tool-design-20261015
fix/hopper-clean-runtime-argv
codex/review-2049-27f1b367
review-fixes/pr-2787-routing
fix/3360-scoped-this-emitters
fix/3367-replaced-global-effects
fix/3838-compact-any-value-items
test/hopper-download-probe-20261015
codex/windows-runtime-creation-red-green-20261015
codex/windows-readonly-root-red-green-20261015
codex/windows-root-share-red-green-20261015
codex/exp-1685-win-mcp-startup
codex/bug-sweep-20261015
investigate/2190-windows-package-phases
investigate/3543-macos-cli-deadline
perf/3281-reverse-tools-feedback-quality
fix/3406-hopper-doctor-fixture
fix/3341-wrapper-require-test-context
fix/3301-owned-uninstall-fixtures
fix/3164-compiler-input-ownership
fix/pcap-cleanup-stdout-newline-20261014
perf/interface-builder-summary
codex/review-fixes-pr-1977
fix/ghidra-effective-jvm-platform
fix/docs-turkish-rust-link-20261013
fix/main-maintenance-ci-20261010
fix/recover-binary-session-process-effects
codex/review-fixes-pr-2457
codex/ci-required-cancellation-gate
fix/main-format-baseline-20261013
fix/setup-preserve-entry-fields
cursor/restore-main-checks-1ffc
fix/ghidra-bridge-hashmap-import
codex/shared-ci-jadx-json-default-20261013
codex/shared-ci-ghidra-test-groups-20261013
codex/shared-ci-java-fixture-format-20261013
codex/fix-intrinsic-invocation-boundaries
test/release-preflight-reusable-jobs
test/js-parameter-provenance-expectation
fix/linux-hopper-verification-current
test/2419-ghidra-elf-probe-size
followup/pr-1544-native-ida
codex/fold-pr-2230-contribution
docs/readme-responsible-use
fix/issue-2064
fix/issue-2050
fix/issue-2053
fix/issue-1968
fix/issue-1945
experiment/codemode-bridge-20261011
fix/tool-effects-negative-wording-1904-20261012
fix/1842-electron-interop-provenance
feat/recover-javascript-multi-input
refactor/website-i18n-prep
fix/nativeaot-pe-virtual-extent
fix/browser-verifier-ci-hang
fix/pe-resources-input-selection
docs/beginner-friendly-guides
docs/beginner-setup-guide
fix/javascript-analysis-worker-1563
fix/1649-ghidra-duplicate-names
fix/javascript-export-keys-integration-1369
test/maintenance-boundary-contracts-20261011
review/javascript-export-keys-1369
review/process-live-progress-1483
test/javascript-1495-growth-ratio
fix/ci-escape-timing
docs/normalized-terminal-geometry-1450
release/6.3.0
archive/release-6.3.0-pr1381
release/6.2.0
fix/electron-preload-dynamic-override
fix/javascript-self-referential-bundler-edges
release/6.1.0
cursor/grok-client-cherry-pick-0b56
feat/signature-posture
feat/dyld-shared-cache
feat/artifact-integrity-policy
feat/pkg-inventory
release/6.0.0
chore/sync-release-5.0.0
release/5.0.0
cursor/cloud-agent-node-787e
fix/javascript-xhr-open-endpoints
review/pr-711-realpath-fallback-wording
cleanup/replay-confinement
codex/test-suite-performance
codex/issue-foundations
codex/address-open-issues-2026-07-20
agent/javascript-vite-rollup-esbuild-adapters
agent/process-cleanup
agent/triage-integration
agent/unknown-commands-registry
feat/mcp-identity-metadata
morlvps/issue_cleanup
codex/installer-redesign
morl/roadmap-investigation-core
morl/roadmap-investigation-docs
release-please--branches--main--components--rea
rea-agents-6.8.0
rea-agents-6.7.0
rea-agents-6.6.0
experiment/atomic-codemode-wait-runtime-20261016
rea-agents-6.5.0
rea-agents-6.4.0
rea-agents-6.3.0
rea-agents-6.2.0
rea-agents-6.1.0
rea-agents-6.0.0
rea-agents-5.0.0
rea-agents-4.1.0
rea-agents-4.0.1
rea-agents-4.0.0
rea-agents-3.2.1
rea-agents-3.2.0
rea-agents-3.1.0
rea-agents-3.0.0
rea-agents-2.7.0
rea-agents-2.6.0
rea-agents-2.5.1
rea-agents-2.5.0
rea-agents-2.4.0
rea-agents-2.3.0
rea-agents-2.2.0
rea-agents-2.1.0
rea-agents-2.0.0
rea-agents-1.7.0
rea-agents-1.6.0
rea-agents-1.5.0
rea-agents-1.4.0
rea-agents-1.3.0
rea-agents-1.2.0
rea-agents-1.1.0
rea-agents-1.0.0
rea-agents-0.5.0
rea-agents-0.4.0
rea-agents-0.3.0
rea-0.2.1
rea-0.2.0
Labels
Clear labels   
area: bridge

   
area: bridge

   
area: build

   
area: build

   
area: build

   
area: cli

   
area: domain

   
area: evidence

   
area: mcp

   
area: native

   
area: providers

   
area: runtime

   
area: server

   
area: setup

   
area: tests

   
bug

   
design

   
documentation

   
enhancement

   
enhancement

   
enhancement

   
experiment

   
help wanted

   
i18n

   
invalid

   
P0: critical

   
P1: high

   
P2: medium

   
P2: medium

   
P2: medium

   
P3: low

   
performance

   
provider-request

   
pull-request
Mirrored from GitHub Pull Request

  
question

   
roadmap

   
triage
No labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
question
roadmap
triage
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5437
No description provided.