[GH-ISSUE #3] Bug: GitHub Releases API not paginated — older releases excluded from daily totals #3

New issue

Open

opened 2026-10-26 22:08:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:08:19 +00:00

Owner

Originally created by @benceruleanlu on GitHub (Nov 9, 2025).
Original GitHub issue: https://github.com/Comfy-Org/download-data/issues/3

Summary
The fetcher calls the Releases API once without pagination, so only the first page (default 30 releases) is processed. Older releases beyond page 1 are ignored, which can undercount and make daily totals appear artificially stable.

Details

Location: scripts/fetch-data.mjs:274–294 (fetchGitHubReleases)
Current: GET https://api.github.com/repos/comfyanonymous/ComfyUI/releases (no per_page, no page loop)
GitHub defaults per_page=30. Popular repos like ComfyUI have many releases; older ones still accrue downloads.
Impact: Any downloads on older releases (outside first page) are omitted, skewing daily deltas. Over time, totals can plateau once interesting releases fall off page 1.

Proposed Fix

Implement pagination:
- Request per_page=100
- Follow Link headers (or increment page starting from 1) until an empty page is returned
- Merge all releases before storing assets into asset_daily_stats
Keep existing PAT/header logic. Consider short delays to respect rate limits.
Log a concise summary: e.g., "Fetched releases across
pages".

Acceptance Criteria

The fetcher retrieves >30 releases when the repo has them (verify by logs and snapshot counts)
Daily deltas change to reflect contributions from older releases where appropriate
No regression in DB schema or API endpoints

References

scripts/fetch-data.mjs:274–294
Delta computation: scripts/fetch-data.mjs:343–357

Originally created by @benceruleanlu on GitHub (Nov 9, 2025). Original GitHub issue: https://github.com/Comfy-Org/download-data/issues/3 Summary The fetcher calls the Releases API once without pagination, so only the first page (default 30 releases) is processed. Older releases beyond page 1 are ignored, which can undercount and make daily totals appear artificially stable. Details - Location: scripts/fetch-data.mjs:274–294 (fetchGitHubReleases) - Current: GET https://api.github.com/repos/comfyanonymous/ComfyUI/releases (no per_page, no page loop) - GitHub defaults per_page=30. Popular repos like ComfyUI have many releases; older ones still accrue downloads. - Impact: Any downloads on older releases (outside first page) are omitted, skewing daily deltas. Over time, totals can plateau once interesting releases fall off page 1. Proposed Fix - Implement pagination: - Request per_page=100 - Follow Link headers (or increment page starting from 1) until an empty page is returned - Merge all releases before storing assets into asset_daily_stats - Keep existing PAT/header logic. Consider short delays to respect rate limits. - Log a concise summary: e.g., "Fetched <N> releases across <P> pages". Acceptance Criteria - The fetcher retrieves >30 releases when the repo has them (verify by logs and snapshot counts) - Daily deltas change to reflect contributions from older releases where appropriate - No regression in DB schema or API endpoints References - scripts/fetch-data.mjs:274–294 - Delta computation: scripts/fetch-data.mjs:343–357

bsakaguchi added the

bug

label

2026-10-26 22:08:19 +00:00

bsakaguchi referenced this issue

2026-10-26 22:08:20 +00:00

[GH-ISSUE #7] Migrate data fetch + storage to Google Cloud (Cloud Run + GCS) #4