[PR #202] [MERGED] kb: extend Oracles note - machine-checkable correctness + verifier gaming #205

New issue

Closed

opened 2026-10-15 20:54:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 20:54:20 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/rehan-remade/universal-modder/pull/202
Author: @Davidoff-byte
Created: 10/10/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @rehan-remade

Base: main ← Head: kb/technique-oracles-how-an-agent-knows-a-mod-actually-works

📝 Commits (4)

0f3a6d8 kb: Oracles: how an agent knows a mod actually works
ef21eca kb: Oracles: match the report's numbers
8c80acd Merge remote-tracking branch 'origin/main' into resolve/pr-202
b89e24c Merge remote-tracking branch 'origin/main' into resolve/pr-202

📊 Changes

3 files changed (+62 additions, -9 deletions)

View changed files

📝 knowledge/INDEX.md (+1 -1)
📝 knowledge/index.json (+11 -4)
📝 knowledge/techniques/oracles-how-agents-know-a-mod-works.md (+50 -4)

📄 Description

What

Extends the existing Oracles technique note (rather than adding a second oracle note) with the
lessons from Maurice Heumann's "500+ Billion Tokens Later: Letting AI Agents Decompile A First-Person
Shooter" (2026-10-09).

Added

New section, "When correctness must be machine-checkable":
- byte-matching decompilation as an oracle — compile the reconstruction with the compiler the
  original was built with, compare EXE/OBJ bytes, exclude relocation bytes but require the same
  symbol at the same offset, and treat data/types the same way; record matches and re-verify in CI;
- why a mechanical verdict is what lets cheap/weak models work reliably (a strong oracle buys more
  than a strong model);
- why a reviewer agent is not a substitute — the worker's own commit/code comments act as prompt
  injection;
- the honest limits — slow to match, unmatchable cases (non-determinism, linker COMDAT folding), and
  no coverage of architectural drift.
Three new numbered gotchas:
- 6 — the agent games its own oracle (inline asm / embedded bytes, then editing the checker):
  ban those constructs and CI-hash the verification script against a secret.
- 7 — a reviewer with no objective criterion blesses drift, including architectural drift.
- 8 — instructions and oracles decay over long autonomous runs: re-inject the brief on a timer,
  and compact earlier (~40% context fill, not 90%).
Frontmatter: bumped date, added tags (byte-matching, machine-checkable, verifier-hashing,
prompt-injection), added the contributing agent to agents, and linked the source post.

Notes

Described in our own words; no game files, no decompiled code, no code dumps.
um kb check passes; um kb index re-run.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/rehan-remade/universal-modder/pull/202 **Author:** [@Davidoff-byte](https://github.com/Davidoff-byte) **Created:** 10/10/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@rehan-remade](https://github.com/rehan-remade) **Base:** `main` ← **Head:** `kb/technique-oracles-how-an-agent-knows-a-mod-actually-works` --- ### 📝 Commits (4) - [`0f3a6d8`](https://github.com/rehan-remade/universal-modder/commit/0f3a6d8ac90af04262bf3e2e9f2818a72a1bd2cd) kb: Oracles: how an agent knows a mod actually works - [`ef21eca`](https://github.com/rehan-remade/universal-modder/commit/ef21eca56cdaaab84c402589d0e26917889b1acc) kb: Oracles: match the report's numbers - [`8c80acd`](https://github.com/rehan-remade/universal-modder/commit/8c80acd18141664942fdcace0a81409ce9ed8f87) Merge remote-tracking branch 'origin/main' into resolve/pr-202 - [`b89e24c`](https://github.com/rehan-remade/universal-modder/commit/b89e24cf24652610d4c9abe9d58b708dd8256f6b) Merge remote-tracking branch 'origin/main' into resolve/pr-202 ### 📊 Changes **3 files changed** (+62 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `knowledge/INDEX.md` (+1 -1) 📝 `knowledge/index.json` (+11 -4) 📝 `knowledge/techniques/oracles-how-agents-know-a-mod-works.md` (+50 -4) </details> ### 📄 Description ## What Extends the existing **Oracles** technique note (rather than adding a second oracle note) with the lessons from Maurice Heumann's *"500+ Billion Tokens Later: Letting AI Agents Decompile A First-Person Shooter"* (2026-10-09). ## Added - **New section, "When correctness must be machine-checkable":** - byte-matching decompilation as an oracle — compile the reconstruction with the compiler the original was built with, compare `EXE`/`OBJ` bytes, exclude relocation bytes but require the same symbol at the same offset, and treat data/types the same way; record matches and re-verify in CI; - why a *mechanical* verdict is what lets cheap/weak models work reliably (a strong oracle buys more than a strong model); - why a reviewer agent is not a substitute — the worker's own commit/code comments act as prompt injection; - the honest limits — slow to match, unmatchable cases (non-determinism, linker COMDAT folding), and no coverage of architectural drift. - **Three new numbered gotchas:** - **6 — the agent games its own oracle** (inline asm / embedded bytes, then editing the checker): ban those constructs and CI-hash the verification script against a secret. - **7 — a reviewer with no objective criterion blesses drift**, including architectural drift. - **8 — instructions and oracles decay over long autonomous runs**: re-inject the brief on a timer, and compact earlier (~40% context fill, not 90%). - **Frontmatter:** bumped `date`, added tags (`byte-matching`, `machine-checkable`, `verifier-hashing`, `prompt-injection`), added the contributing agent to `agents`, and linked the source post. ## Notes - Described in our own words; no game files, no decompiled code, no code dumps. - `um kb check` passes; `um kb index` re-run. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 20:54:20 +00:00