[PR #203] [MERGED] perf: fix CPU spike — add early $match before $unwind in CNRepos pipeline #201

New issue

Closed

opened 2026-10-27 21:10:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:10:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/Comfy-PR/pull/203
Author: @snomiao
Created: 4/24/2026
Status: ✅ Merged
Merged: 4/24/2026
Merged by: @snomiao

Base: main ← Head: perf/fix-cnrepos-cpu-spike

📝 Commits (4)

3612970 perf: fix CPU spike — add early $match before $unwind in CNRepos pipeline
4ea4709 fix: remove pullRequestMessage index — values exceed B-tree key size limit
1f7c609 fix: surface index build failures via console.error instead of silencing
79c13ba fix: use non-sparse index on GithubActionUpdateTask.error

📊 Changes

3 files changed (+17 additions, -0 deletions)

View changed files

📝 src/CNRepos.ts (+8 -0)
📝 src/GithubActionUpdateTask/GithubActionUpdateTask.ts (+6 -0)
📝 src/analyzePullsStatus.ts (+3 -0)

📄 Description

Summary

Investigated 2026-04-23T22:12Z Atlas CPU spike (98.5% CPU on comfyorg-prod-efe3368)
Query Profiler confirmed: comfy-pr.CNRepos reads → 737 ops × avg 5.49 min = 67.44 CPU-hours in 10.5h window; Uses Index: No
Root cause: baseCRPullStatusPipeline() scanned all CNRepos docs, $unwinded every crPulls.data entry, then filtered — processing everything before discarding irrelevant documents

Fixes:

Add early $match { crPulls.data: { $elemMatch: { "comments.state": "ok" } } } before $unwind to skip repos with no fetched PR comments
Add supporting multikey index on crPulls.data.comments.state (scalar Task field — using comments.data would fail with "Cannot index parallel arrays")
Add sparse indexes on GithubActionUpdateTask.error and .pullRequestMessage (Performance Advisor flagged 4310-doc regex scans at 0.42/hr)

Test plan

Monitor Atlas Query Profiler after deploy — CNRepos read avg execution time should drop from 5.49 min toward <1s
Verify idx_crpulls_comments_state index builds successfully in Atlas (no "parallel arrays" error)
Dashboard /details page still renders correctly with the filtered pipeline
analyzePullsStatus() still returns same data (the post-$unwind .match({ "crPulls.data.comments.data": { $exists: true } }) is kept as safety net)

Investigation Evidence

Atlas Query Profiler (2026-04-23 22:00 JST → 2026-04-24 08:30 JST):

Collection	Op	Count	Mean	Sum
CNRepos	read	737	5.49 min	67.44 hr
CNRepos	write	108	319 ms	34.50 s

Actual query logged: aggregate CNRepos → $set → $unwind "$crPulls.data" → $match → ... (Num Yields: 5,043, Uses Index: No)

Reviewed by Gemini 2.5 Flash ✅, Gemini 2.5 Pro ✅ (flagged parallel-array index issue → fixed), Codex gpt-5.4 ✅

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/Comfy-PR/pull/203 **Author:** [@snomiao](https://github.com/snomiao) **Created:** 4/24/2026 **Status:** ✅ Merged **Merged:** 4/24/2026 **Merged by:** [@snomiao](https://github.com/snomiao) **Base:** `main` ← **Head:** `perf/fix-cnrepos-cpu-spike` --- ### 📝 Commits (4) - [`3612970`](https://github.com/Comfy-Org/Comfy-PR/commit/361297014a40176687a7ec3a6fb4dddd537b270e) perf: fix CPU spike — add early $match before $unwind in CNRepos pipeline - [`4ea4709`](https://github.com/Comfy-Org/Comfy-PR/commit/4ea4709101a755a8c5ffb3e22109c0e2a0fe4320) fix: remove pullRequestMessage index — values exceed B-tree key size limit - [`1f7c609`](https://github.com/Comfy-Org/Comfy-PR/commit/1f7c6096a458ab5ab7adf9573ed5196709c8e0df) fix: surface index build failures via console.error instead of silencing - [`79c13ba`](https://github.com/Comfy-Org/Comfy-PR/commit/79c13bab9def62e9dc0483a9cd4cde8a72989d4c) fix: use non-sparse index on GithubActionUpdateTask.error ### 📊 Changes **3 files changed** (+17 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `src/CNRepos.ts` (+8 -0) 📝 `src/GithubActionUpdateTask/GithubActionUpdateTask.ts` (+6 -0) 📝 `src/analyzePullsStatus.ts` (+3 -0) </details> ### 📄 Description ## Summary - Investigated 2026-04-23T22:12Z Atlas CPU spike (98.5% CPU on comfyorg-prod-efe3368) - Query Profiler confirmed: `comfy-pr.CNRepos` reads → 737 ops × avg **5.49 min** = **67.44 CPU-hours** in 10.5h window; `Uses Index: No` - Root cause: `baseCRPullStatusPipeline()` scanned all CNRepos docs, `$unwind`ed every `crPulls.data` entry, then filtered — processing everything before discarding irrelevant documents **Fixes:** - Add early `$match { crPulls.data: { $elemMatch: { "comments.state": "ok" } } }` before `$unwind` to skip repos with no fetched PR comments - Add supporting multikey index on `crPulls.data.comments.state` (scalar `Task` field — using `comments.data` would fail with "Cannot index parallel arrays") - Add sparse indexes on `GithubActionUpdateTask.error` and `.pullRequestMessage` (Performance Advisor flagged 4310-doc regex scans at 0.42/hr) ## Test plan - [ ] Monitor Atlas Query Profiler after deploy — CNRepos read avg execution time should drop from 5.49 min toward <1s - [ ] Verify `idx_crpulls_comments_state` index builds successfully in Atlas (no "parallel arrays" error) - [ ] Dashboard `/details` page still renders correctly with the filtered pipeline - [ ] `analyzePullsStatus()` still returns same data (the post-`$unwind` `.match({ "crPulls.data.comments.data": { $exists: true } })` is kept as safety net) ## Investigation Evidence Atlas Query Profiler (2026-04-23 22:00 JST → 2026-04-24 08:30 JST): | Collection | Op | Count | Mean | Sum | |---|---|---|---|---| | CNRepos | read | 737 | **5.49 min** | **67.44 hr** | | CNRepos | write | 108 | 319 ms | 34.50 s | Actual query logged: `aggregate CNRepos → $set → $unwind "$crPulls.data" → $match → ...` (Num Yields: 5,043, Uses Index: No) Reviewed by Gemini 2.5 Flash ✅, Gemini 2.5 Pro ✅ (flagged parallel-array index issue → fixed), Codex gpt-5.4 ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:10:34 +00:00

bsakaguchi referenced this issue

2026-10-27 21:10:34 +00:00

[PR #201] [MERGED] perf: performance fixes, bugcop 404 handling, and createIndex docs #202

bsakaguchi closed this issue

2026-10-27 21:10:34 +00:00