[PR #89] feat: add test evidence checker for PR submissions (#61) #90

New issue

Open

opened 2026-10-27 21:09:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:09:28 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/Comfy-PR/pull/89
Author: @snomiao
Created: 10/20/2025
Status: 🔄 Open

Base: main ← Head: sno-test-evidence

📝 Commits (1)

05f4e29 feat(gh-test-evidence): add test evidence checker for PR submissions (#61)

📊 Changes

3 files changed (+602 additions, -0 deletions)

View changed files

➕ app/tasks/gh-test-evidence/gh-test-evidence.spec.ts (+283 -0)
➕ app/tasks/gh-test-evidence/gh-test-evidence.ts (+314 -0)
📝 app/tasks/run-gh-tasks.ts (+5 -0)

📄 Description

Summary

Implements automated test evidence checking for PRs in desktop and ComfyUI repos, solving issue #61 in a smarter way.

Changes

✅ Created app/tasks/gh-test-evidence/gh-test-evidence.ts task
✅ Scans open PRs in Comfy-Org/desktop and comfyanonymous/ComfyUI
✅ Uses GPT-4o-mini (cheaper, faster) to analyze PR bodies
✅ Detects test explanations, screenshots, and videos
✅ Posts warning comments when evidence is missing
✅ Auto-updates comments when PR changes
✅ Deletes comments when all evidence is present
✅ Follows ComfyUI_frontend workflow message format
✅ Added comprehensive test file

Smart Improvements

Efficient AI model: Uses GPT-4o-mini instead of GPT-4o for faster, cheaper analysis
Clean architecture: Follows existing task patterns (coreping, gh-bounty)
Idempotent: Re-analyzes only when PR updates
Smart comment management: Updates existing comments instead of spamming
Database tracking: Uses MongoDB to track task state
Type-safe: Full TypeScript with Zod validation
Bot marker: Uses HTML comment marker for identifying bot comments

Testing

Added gh-test-evidence.spec.ts with test structure
Follows project test patterns
Tests cover: draft PRs, missing evidence, complete evidence, comment updates

Workflow Integration

Added to app/tasks/run-gh-tasks.ts to run on schedule with other GitHub tasks.

Closes #61

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/Comfy-PR/pull/89 **Author:** [@snomiao](https://github.com/snomiao) **Created:** 10/20/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `sno-test-evidence` --- ### 📝 Commits (1) - [`05f4e29`](https://github.com/Comfy-Org/Comfy-PR/commit/05f4e291b600794f58983299e843879212c8257f) feat(gh-test-evidence): add test evidence checker for PR submissions (#61) ### 📊 Changes **3 files changed** (+602 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `app/tasks/gh-test-evidence/gh-test-evidence.spec.ts` (+283 -0) ➕ `app/tasks/gh-test-evidence/gh-test-evidence.ts` (+314 -0) 📝 `app/tasks/run-gh-tasks.ts` (+5 -0) </details> ### 📄 Description ## Summary Implements automated test evidence checking for PRs in desktop and ComfyUI repos, solving issue #61 in a smarter way. ## Changes - ✅ Created `app/tasks/gh-test-evidence/gh-test-evidence.ts` task - ✅ Scans open PRs in `Comfy-Org/desktop` and `comfyanonymous/ComfyUI` - ✅ Uses GPT-4o-mini (cheaper, faster) to analyze PR bodies - ✅ Detects test explanations, screenshots, and videos - ✅ Posts warning comments when evidence is missing - ✅ Auto-updates comments when PR changes - ✅ Deletes comments when all evidence is present - ✅ Follows ComfyUI_frontend workflow message format - ✅ Added comprehensive test file ## Smart Improvements 1. **Efficient AI model**: Uses GPT-4o-mini instead of GPT-4o for faster, cheaper analysis 2. **Clean architecture**: Follows existing task patterns (coreping, gh-bounty) 3. **Idempotent**: Re-analyzes only when PR updates 4. **Smart comment management**: Updates existing comments instead of spamming 5. **Database tracking**: Uses MongoDB to track task state 6. **Type-safe**: Full TypeScript with Zod validation 7. **Bot marker**: Uses HTML comment marker for identifying bot comments ## Testing - Added `gh-test-evidence.spec.ts` with test structure - Follows project test patterns - Tests cover: draft PRs, missing evidence, complete evidence, comment updates ## Workflow Integration Added to `app/tasks/run-gh-tasks.ts` to run on schedule with other GitHub tasks. Closes #61 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:09:28 +00:00

bsakaguchi referenced this issue

2026-10-27 21:09:28 +00:00

[PR #90] [MERGED] feat: add MSW for mocking GitHub API in tests #91