[PR #401] [MERGED] feat(cursor-review): record the direct judge's token usage #402

New issue

Closed

opened 2026-10-26 22:39:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:39:25 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/github-workflows/pull/401
Author: @mattmillerai
Created: 10/9/2026
Status: ✅ Merged
Merged: 10/9/2026
Merged by: @mattmillerai

Base: main ← Head: matt/judge-direct-usage

📝 Commits (2)

a4b349b feat(cursor-review): record the direct judge's token usage
a2fb19e fix(cursor-review): time-box judge usage steps and flag a failed usage upload

📊 Changes

4 files changed (+227 additions, -4 deletions)

View changed files

📝 .github/cursor-review/README.md (+1 -1)
📝 .github/cursor-review/tests/test_workflow_judge_direct.py (+105 -2)
📝 .github/workflows/cursor-review.yml (+120 -0)
📝 docs/callers/cursor-review.md (+1 -1)

📄 Description

ELI-5

Each direct-Anthropic review cell already saves how many tokens it used, so you can work out what it cost. The direct judge from #399 doesn't save anything. This PR makes the judge save the same token counts, so its cost can be measured too.

Summary

judge-direct gains the cells' two accounting steps, Record token usage and Upload usage artifact.

Same script as the cells. The record script matches the cells' byte for byte except for the result path. Run judge (direct) already writes Claude Code's result JSON to /tmp/judge-claude-result.json, and the script reads it from there. A test pins the match, so the two scripts can't drift apart.
- The script copies only integer token counts and model ids. Nothing model-written goes in, and no prices.
- If the judge left no usable result, the record says measured: false instead of a guessed zero.
Artifact: usage-direct-judge-<model>, kept for 30 days.
- It belongs to the cells' usage-direct-<review_type>-<model> family, with review type judge.
- The name doesn't match findings-* and isn't cursor-review-judge-direct, so neither consolidate nor the hand-back reads it.
When it runs: whenever the judge step ran. The condition is always() && steps.judge_direct.outcome != 'skipped', and Run judge (direct) gains id: judge_direct. A judge killed by its time cap is still recorded, as unmeasured.
Can't affect the review:
- Both steps are continue-on-error.
- They run after the review's own upload, so accounting can neither fail nor delay the hand-off.
- Neither step references ANTHROPIC_API_KEY.
Docs: the judge_direct_model rows in .github/cursor-review/README.md and docs/callers/cursor-review.md now mention the artifact.

Tests

New JudgeTokenUsageTest in test_workflow_judge_direct.py. It checks:
- Identity: the judge's script equals the cells' script with only the result path changed, and that path is where Run judge (direct) writes.
- Executed over a judge result:
  - the counts are copied, with review_type: judge;
  - the model text and the CLI's cost figure are dropped;
  - with no result file at all, the record is unmeasured.
- The if:, evaluated for success, failure, cancelled and skipped.
- Safety: both steps are best effort and key-free, the artifact name is right, and they come last in the job.
test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python now expects three python3 -I - scripts, all with -I.
Mutation check: pointing the judge's record step back at the cells' result path fails two of the new tests.
python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py': 1334 tests, OK.
python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py': 551 tests, OK (43 skipped).
actionlint: 80 findings on main and 80 on this branch, so none are new. They are the existing job.workflow_sha and SC2018/SC2019 notes.
Not run: a live round with the direct judge. No caller has enabled it yet.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/github-workflows/pull/401 **Author:** [@mattmillerai](https://github.com/mattmillerai) **Created:** 10/9/2026 **Status:** ✅ Merged **Merged:** 10/9/2026 **Merged by:** [@mattmillerai](https://github.com/mattmillerai) **Base:** `main` ← **Head:** `matt/judge-direct-usage` --- ### 📝 Commits (2) - [`a4b349b`](https://github.com/Comfy-Org/github-workflows/commit/a4b349bdc246cbe23ab65c3012fa09b9be33e87f) feat(cursor-review): record the direct judge's token usage - [`a2fb19e`](https://github.com/Comfy-Org/github-workflows/commit/a2fb19e7e77f8ee1ebf99a539969a71323c28bb5) fix(cursor-review): time-box judge usage steps and flag a failed usage upload ### 📊 Changes **4 files changed** (+227 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `.github/cursor-review/README.md` (+1 -1) 📝 `.github/cursor-review/tests/test_workflow_judge_direct.py` (+105 -2) 📝 `.github/workflows/cursor-review.yml` (+120 -0) 📝 `docs/callers/cursor-review.md` (+1 -1) </details> ### 📄 Description ## ELI-5 Each direct-Anthropic review cell already saves how many tokens it used, so you can work out what it cost. The direct judge from #399 doesn't save anything. This PR makes the judge save the same token counts, so its cost can be measured too. ## Summary `judge-direct` gains the cells' two accounting steps, `Record token usage` and `Upload usage artifact`. - **Same script as the cells.** The record script matches the cells' byte for byte except for the result path. `Run judge (direct)` already writes Claude Code's result JSON to `/tmp/judge-claude-result.json`, and the script reads it from there. A test pins the match, so the two scripts can't drift apart. - The script copies only integer token counts and model ids. Nothing model-written goes in, and no prices. - If the judge left no usable result, the record says `measured: false` instead of a guessed zero. - **Artifact:** `usage-direct-judge-<model>`, kept for 30 days. - It belongs to the cells' `usage-direct-<review_type>-<model>` family, with review type `judge`. - The name doesn't match `findings-*` and isn't `cursor-review-judge-direct`, so neither `consolidate` nor the hand-back reads it. - **When it runs:** whenever the judge step ran. The condition is `always() && steps.judge_direct.outcome != 'skipped'`, and `Run judge (direct)` gains `id: judge_direct`. A judge killed by its time cap is still recorded, as unmeasured. - **Can't affect the review:** - Both steps are `continue-on-error`. - They run after the review's own upload, so accounting can neither fail nor delay the hand-off. - Neither step references `ANTHROPIC_API_KEY`. - **Docs:** the `judge_direct_model` rows in `.github/cursor-review/README.md` and `docs/callers/cursor-review.md` now mention the artifact. ## Tests - New `JudgeTokenUsageTest` in `test_workflow_judge_direct.py`. It checks: - **Identity:** the judge's script equals the cells' script with only the result path changed, and that path is where `Run judge (direct)` writes. - **Executed** over a judge result: - the counts are copied, with `review_type: judge`; - the model text and the CLI's cost figure are dropped; - with no result file at all, the record is unmeasured. - **The `if:`**, evaluated for `success`, `failure`, `cancelled` and `skipped`. - **Safety:** both steps are best effort and key-free, the artifact name is right, and they come last in the job. - `test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python` now expects three `python3 -I -` scripts, all with `-I`. - **Mutation check:** pointing the judge's record step back at the cells' result path fails two of the new tests. - `python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'`: 1334 tests, OK. - `python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py'`: 551 tests, OK (43 skipped). - actionlint: 80 findings on `main` and 80 on this branch, so none are new. They are the existing `job.workflow_sha` and SC2018/SC2019 notes. - Not run: a live round with the direct judge. No caller has enabled it yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:39:25 +00:00

bsakaguchi referenced this issue

2026-10-26 22:39:25 +00:00

[PR #404] [MERGED] feat(cursor-review): name the Anthropic lane's failure cause, gated and key-screened #404

bsakaguchi closed this issue

2026-10-26 22:39:26 +00:00

bsakaguchi referenced this issue

2026-10-26 22:39:26 +00:00

[PR #402] [MERGED] feat(cursor-review): say why a direct-API cell failed instead of discarding it #405