[GH-ISSUE #7] Migrate data fetch + storage to Google Cloud (Cloud Run + GCS) #4

New issue

Open

opened 2026-10-26 22:08:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:08:20 +00:00

Owner

Originally created by @benceruleanlu on GitHub (Nov 9, 2025).
Original GitHub issue: https://github.com/Comfy-Org/download-data/issues/7

Priority
High — move off GitHub Actions + local SQLite to Google Cloud.

Background

Fetcher runs in GitHub Actions and writes data/downloads.db (SQLite).
- Workflow: .github/workflows/fetch-data.yml
- Fetcher: scripts/fetch-data.mjs
- API reads SQLite: app/api/downloads/daily/route.ts

Goals

Run data fetching in Google Cloud on a daily schedule.
Store consumable data in GCS (JSON) so the dashboard can read it without SQLite.
Keep daily bucketing consistent with DAILY_CUTOFF_UTC and current logic.

Proposal (Option A — recommended, minimal code risk)

Cloud Run Job for the fetcher (containerized):
- Use Secret Manager for PAT (GitHub token).
- Trigger daily via Cloud Scheduler at 10:49 UTC (matches DAILY_CUTOFF_UTC).
- Fetch with pagination (see issue #3) and apply portable/stable filters (issue #4).
- Emit artifacts to GCS bucket, e.g.:
  - downloads/daily_summary.json
  - downloads/backfill_events.json
  - (optional) downloads/asset_daily_stats.jsonl for debugging.
Next.js API routes read from GCS JSON instead of SQLite.
- Replace better-sqlite3 in API with simple fetch/parsing of GCS JSON.
- Leave SQLite usage internal to the job only (or remove it later).

Alternatives

Option B: Cloud SQL (Postgres) and migrate SQL (replace SQLite-specific syntax).
Option C: BigQuery for analytics; precompute and cache in GCS for the UI.

Acceptance Criteria

Daily run on Cloud Run Job updates GCS JSON at/after cutoff time.
Dashboard loads from GCS-backed API and matches local results (aside from pagination/filters fixes).
GitHub Actions workflow for fetching is removed or disabled.
Secrets managed via Secret Manager; no tokens in repo.

Tasks

Add Dockerfile + containerize fetcher
Implement release pagination (#3)
Add portable/stable filters behind env flags (#4)
Emit JSON artifacts to GCS (daily_summary, backfill_events)
Add Cloud Run Job, Scheduler, and Secret Manager wiring
Refactor Next.js API to read from GCS JSON
Disable/remove .github/workflows/fetch-data.yml
Documentation: deployment, env vars, rollback

References

Fetcher + cutoff: scripts/fetch-data.mjs (getBucketDate, DAILY_CUTOFF_UTC)
Current API: app/api/downloads/daily/route.ts
Current workflow: .github/workflows/fetch-data.yml

Originally created by @benceruleanlu on GitHub (Nov 9, 2025). Original GitHub issue: https://github.com/Comfy-Org/download-data/issues/7 Priority High — move off GitHub Actions + local SQLite to Google Cloud. Background - Fetcher runs in GitHub Actions and writes `data/downloads.db` (SQLite). - Workflow: `.github/workflows/fetch-data.yml` - Fetcher: `scripts/fetch-data.mjs` - API reads SQLite: `app/api/downloads/daily/route.ts` Goals - Run data fetching in Google Cloud on a daily schedule. - Store consumable data in GCS (JSON) so the dashboard can read it without SQLite. - Keep daily bucketing consistent with `DAILY_CUTOFF_UTC` and current logic. Proposal (Option A — recommended, minimal code risk) - Cloud Run Job for the fetcher (containerized): - Use Secret Manager for `PAT` (GitHub token). - Trigger daily via Cloud Scheduler at 10:49 UTC (matches `DAILY_CUTOFF_UTC`). - Fetch with pagination (see issue #3) and apply portable/stable filters (issue #4). - Emit artifacts to GCS bucket, e.g.: - `downloads/daily_summary.json` - `downloads/backfill_events.json` - (optional) `downloads/asset_daily_stats.jsonl` for debugging. - Next.js API routes read from GCS JSON instead of SQLite. - Replace `better-sqlite3` in API with simple fetch/parsing of GCS JSON. - Leave SQLite usage internal to the job only (or remove it later). Alternatives - Option B: Cloud SQL (Postgres) and migrate SQL (replace SQLite-specific syntax). - Option C: BigQuery for analytics; precompute and cache in GCS for the UI. Acceptance Criteria - Daily run on Cloud Run Job updates GCS JSON at/after cutoff time. - Dashboard loads from GCS-backed API and matches local results (aside from pagination/filters fixes). - GitHub Actions workflow for fetching is removed or disabled. - Secrets managed via Secret Manager; no tokens in repo. Tasks - [ ] Add Dockerfile + containerize fetcher - [ ] Implement release pagination (#3) - [ ] Add portable/stable filters behind env flags (#4) - [ ] Emit JSON artifacts to GCS (daily_summary, backfill_events) - [ ] Add Cloud Run Job, Scheduler, and Secret Manager wiring - [ ] Refactor Next.js API to read from GCS JSON - [ ] Disable/remove `.github/workflows/fetch-data.yml` - [ ] Documentation: deployment, env vars, rollback References - Fetcher + cutoff: `scripts/fetch-data.mjs` (getBucketDate, DAILY_CUTOFF_UTC) - Current API: `app/api/downloads/daily/route.ts` - Current workflow: `.github/workflows/fetch-data.yml`

bsakaguchi added the

enhancement

label

2026-10-26 22:08:20 +00:00