morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8084] [Design] Skill: keep per-target guidance out of SKILL.md, in condition-named reference pages #4667

New issue

Open

opened 2026-10-31 00:52:12 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 00:52:12 +00:00

Owner

Originally created by @Entrpi on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8084

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Should SKILL.md follow progressive disclosure: routing and cross-cutting rules always loaded, and per-target guidance in pages that SKILL.md names, with when to read each? And should future guidance follow one placement rule: put it on the narrowest layer that is loaded when it changes what the agent does?

SKILL.md loads about 8,300 tokens on every activation. About 3,000 of them are single-format entries that apply only when that format is supplied, and the file exceeds both published skill guides (under 500 lines; under 5,000 tokens). The guides' answer is progressive disclosure: SKILL.md routes, and detail loads when a task needs it (specification, authoring guide). A word-for-word regrouping on that model cuts skill-plus-page tokens by 59–87% on five of six tasks and removes no guidance. The open risk is whether agents still open the right page; the verify:agent scenarios below would check that.

Context and evidence

Observed at main 317bf3fe2 with four offline counters: DeepSeek-V4 BPE, Qwen2 BPE, GPT-2 BPE and REA's tokenx. Claude's tokenizer is not available offline, so treat absolute numbers as proxies. The ratios hold under every counter.

SKILL.md is 508 lines and about 8,300 tokens (tokenx about 9,100). Every skill activation loads all of it. The routing section is 70% of the file. Its 21 single-format inspector entries cost about 3,000 tokens, and each matters only when that format is supplied.
references/native-and-artifacts.md is about 10,500 tokens over 714 lines and has no contents list. SKILL.md sends native, archive, extraction and ESP tasks to it. The raw-image paragraph sits near line 600, inside "Packages and extraction", and ESP images is the second-last section.
Many tools are described in up to four places, in different words: the SKILL.md entry, a reference section, the tool description and a docs page. One copy has already drifted. SKILL.md says to set REA_CAPTURE_DECODER_OLD_GENERATION_MIB where REA starts, but the decoder's refusal names capture.decoder_old_generation_mib in-band, and that setting can be changed at runtime.

Skill plus pages per task, today and after a prototype regrouping. The regrouping moves existing text verbatim and uses a 115-line SKILL.md draft:

Task	Today	After
Cortex-M raw dump	18,790	2,416
ESP32 app image	18,790	2,416
JavaScript bundle, trace a feature	11,394	4,690
APK code and manifest	9,192	2,529
Failed connection	11,732	6,053
Native function with Ghidra	18,790	6,450

Tool definitions are a separate cost and do not change here: about 64,400 tokens per request for 233 definitions in clients that load them all, and names only in Claude Code with tool search.

Expected interaction with targets

No target interaction (documentation or usability only)

Boundary details

No response

Options and tradeoffs

Proposed, additive only:

SKILL.md keeps scope, the connection and safety rules, and an 11-line routing list. Each line names the condition and the page.
All seven reference paths stay. Four pages are added: firmware-and-raw-images.md, packages-and-archives.md, format-inspectors.md and recorded-captures.md.
Duplicate inspector entries merge into their existing sections.
SKILL.md links every page directly, pages do not link to each other, and every page over 100 lines starts with a contents list.
The new pages join MANAGED_SKILL_FILES and the setup tests. No page is renamed or removed, so setup and uninstall ownership does not change.

Cost: four new managed files, a few hard-coded lists in the setup tests, and one path in a conformance test. The main risk is an agent that skips a breadcrumb. The safety rules stay in SKILL.md and in the server instructions, and call-time facts stay in tool descriptions.

Alternatives: (a) only compress the SKILL.md routing list, without splitting pages; (b) drop the per-format entries from the skill and rely on tool descriptions and docs. Option (b) is cheaper, but CLI-only agents then lose that guidance.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

npm run docs:check and the setup tests pass. The skill-reference check runs on the installed copy.
verify:agent passes before and after, with two added scenarios: a raw Cortex-M dump and an ESP32 image.

Questions:

Do CLI-only agents need per-format text in the skill, or are rea <command> --help and the docs enough?
One format page, or two (PE/ELF tables; bytecode and metadata)?
Which agent-evaluation scenarios would you accept as evidence?

Sequencing: after #7951 and #7486, which edit native-and-artifacts.md.

References

Agent Skills specification, "Progressive disclosure"
Skill authoring best practices, "Progressive disclosure patterns"

Originally created by @Entrpi on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8084 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Should SKILL.md follow progressive disclosure: routing and cross-cutting rules always loaded, and per-target guidance in pages that SKILL.md names, with when to read each? And should future guidance follow one placement rule: put it on the narrowest layer that is loaded when it changes what the agent does? SKILL.md loads about 8,300 tokens on every activation. About 3,000 of them are single-format entries that apply only when that format is supplied, and the file exceeds both published skill guides (under 500 lines; under 5,000 tokens). The guides' answer is progressive disclosure: SKILL.md routes, and detail loads when a task needs it ([specification](https://agentskills.io/specification#progressive-disclosure), [authoring guide](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#progressive-disclosure-patterns)). A word-for-word regrouping on that model cuts skill-plus-page tokens by 59–87% on five of six tasks and removes no guidance. The open risk is whether agents still open the right page; the `verify:agent` scenarios below would check that. ### Context and evidence Observed at main `317bf3fe2` with four offline counters: DeepSeek-V4 BPE, Qwen2 BPE, GPT-2 BPE and REA's `tokenx`. Claude's tokenizer is not available offline, so treat absolute numbers as proxies. The ratios hold under every counter. - SKILL.md is 508 lines and about 8,300 tokens (`tokenx` about 9,100). Every skill activation loads all of it. The routing section is 70% of the file. Its 21 single-format inspector entries cost about 3,000 tokens, and each matters only when that format is supplied. - `references/native-and-artifacts.md` is about 10,500 tokens over 714 lines and has no contents list. SKILL.md sends native, archive, extraction and ESP tasks to it. The raw-image paragraph sits near line 600, inside "Packages and extraction", and ESP images is the second-last section. - Many tools are described in up to four places, in different words: the SKILL.md entry, a reference section, the tool description and a docs page. One copy has already drifted. SKILL.md says to set `REA_CAPTURE_DECODER_OLD_GENERATION_MIB` where REA starts, but the decoder's refusal names `capture.decoder_old_generation_mib` in-band, and that setting can be changed at runtime. Skill plus pages per task, today and after a prototype regrouping. The regrouping moves existing text verbatim and uses a 115-line SKILL.md draft: | Task | Today | After | | --- | ---: | ---: | | Cortex-M raw dump | 18,790 | 2,416 | | ESP32 app image | 18,790 | 2,416 | | JavaScript bundle, trace a feature | 11,394 | 4,690 | | APK code and manifest | 9,192 | 2,529 | | Failed connection | 11,732 | 6,053 | | Native function with Ghidra | 18,790 | 6,450 | Tool definitions are a separate cost and do not change here: about 64,400 tokens per request for 233 definitions in clients that load them all, and names only in Claude Code with tool search. ### Expected interaction with targets No target interaction (documentation or usability only) ### Boundary details _No response_ ### Options and tradeoffs Proposed, additive only: - SKILL.md keeps scope, the connection and safety rules, and an 11-line routing list. Each line names the condition and the page. - All seven reference paths stay. Four pages are added: `firmware-and-raw-images.md`, `packages-and-archives.md`, `format-inspectors.md` and `recorded-captures.md`. - Duplicate inspector entries merge into their existing sections. - SKILL.md links every page directly, pages do not link to each other, and every page over 100 lines starts with a contents list. - The new pages join `MANAGED_SKILL_FILES` and the setup tests. No page is renamed or removed, so setup and uninstall ownership does not change. Cost: four new managed files, a few hard-coded lists in the setup tests, and one path in a conformance test. The main risk is an agent that skips a breadcrumb. The safety rules stay in SKILL.md and in the server instructions, and call-time facts stay in tool descriptions. Alternatives: (a) only compress the SKILL.md routing list, without splitting pages; (b) drop the per-format entries from the skill and rely on tool descriptions and docs. Option (b) is cheaper, but CLI-only agents then lose that guidance. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions - `npm run docs:check` and the setup tests pass. The skill-reference check runs on the installed copy. - `verify:agent` passes before and after, with two added scenarios: a raw Cortex-M dump and an ESP32 image. Questions: 1. Do CLI-only agents need per-format text in the skill, or are `rea <command> --help` and the docs enough? 2. One format page, or two (PE/ELF tables; bytecode and metadata)? 3. Which agent-evaluation scenarios would you accept as evidence? Sequencing: after #7951 and #7486, which edit `native-and-artifacts.md`. ### References - [Agent Skills specification](https://agentskills.io/specification), "Progressive disclosure" - [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices), "Progressive disclosure patterns"

bsakaguchi commented

2026-10-31 00:52:14 +00:00

Author

Owner

@aakash-env commented on GitHub (Oct 30, 2026):

@Entrpi @baron @monomyth kindly assign this issue to me

@aakash-env commented on GitHub (Oct 30, 2026): @Entrpi @baron @monomyth kindly assign this issue to me

bsakaguchi commented

2026-10-31 00:52:14 +00:00

Author

Owner

@bonro commented on GitHub (Oct 30, 2026):

We offer the full series of GPT models at 0.2x official pricing and the full series of Claude models at 0.3x official pricing, available now at lanmu1313.top.

@bonro commented on GitHub (Oct 30, 2026): We offer the full series of GPT models at 0.2x official pricing and the full series of Claude models at 0.3x official pricing, available now at lanmu1313.top.

bsakaguchi commented

2026-10-31 00:52:14 +00:00

Author

Owner

@pablodiazjorge commented on GitHub (Oct 30, 2026):

One data point on the open risk, whether agents open the page SKILL.md names, from a preregistered study I ran on Windows with rea 6.4.0 and its skill:

120 Claude Code runs (claude-sonnet-5-5, medium effort) locating functions in symbol-less PE builds of four open-source programs, with Ghidra 12.1.4. The body of SKILL.md went into the system prompt and its references/ files were readable, so loading the skill was not left to the client.
SKILL.md's "Read only the relevant guide" sent native binaries to references/native-and-artifacts.md. No run opened it, or any other reference page (0 of 120).
Accuracy was 87 to 92 % with or without the skill, so these tasks could be solved without the page. The result says agents don't follow a breadcrumb they don't need; it doesn't show they would skip one they do.
Cost: the official skill in the prompt made no measurable difference against no skill (Hodges-Lehmann cost ratio 0.98, 95 % interval 0.82 to 1.16, 30 tasks with 2 runs each). About 70 % of the cost was cache writes of the growing context, so the always-loaded skill was a small share in Claude Code; a client without prompt caching would weigh it differently.

So for the regrouping, a scenario where the page is actually required, like your Cortex-M and ESP32 cases, looks like the right check, and counting page reads in the transcripts would show whether agents follow the routing line.

Writeup, data and preregistration: https://github.com/pablodiazjorge/rea-lab/tree/main/writeups/2026-10-study-5

@pablodiazjorge commented on GitHub (Oct 30, 2026): One data point on the open risk, whether agents open the page SKILL.md names, from a preregistered study I ran on Windows with rea 6.4.0 and its skill: - 120 Claude Code runs (`claude-sonnet-5-5`, medium effort) locating functions in symbol-less PE builds of four open-source programs, with Ghidra 12.1.4. The body of SKILL.md went into the system prompt and its `references/` files were readable, so loading the skill was not left to the client. - SKILL.md's "Read only the relevant guide" sent native binaries to `references/native-and-artifacts.md`. No run opened it, or any other reference page (0 of 120). - Accuracy was 87 to 92 % with or without the skill, so these tasks could be solved without the page. The result says agents don't follow a breadcrumb they don't need; it doesn't show they would skip one they do. - Cost: the official skill in the prompt made no measurable difference against no skill (Hodges-Lehmann cost ratio 0.98, 95 % interval 0.82 to 1.16, 30 tasks with 2 runs each). About 70 % of the cost was cache writes of the growing context, so the always-loaded skill was a small share in Claude Code; a client without prompt caching would weigh it differently. So for the regrouping, a scenario where the page is actually required, like your Cortex-M and ESP32 cases, looks like the right check, and counting page reads in the transcripts would show whether agents follow the routing line. Writeup, data and preregistration: https://github.com/pablodiazjorge/rea-lab/tree/main/writeups/2026-10-study-5

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4667

No description provided.

Rows
Columns