[PR #203] [MERGED] chore(scripts): build an observed OpenAPI document of the Boosty API #202

New issue

Closed

opened 2026-10-13 19:48:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/203
Author: @Glitchy-Sheep
Created: 9/22/2026
Status: ✅ Merged
Merged: 9/22/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: feat/api-schema-snapshot

📝 Commits (3)

fba0390 [scripts] Infer an OpenAPI document from Boosty answers
f6ffb0f [scripts] Add task api:schema that writes docs/api/boosty-api.yaml
a77594c [scripts] Name the api_schema modules and functions after what they do

📊 Changes

13 files changed (+1914 additions, -1 deletions)

View changed files

📝 Taskfile.yml (+15 -0)
➕ docs/api/README.md (+27 -0)
➕ docs/api/boosty-api.yaml (+883 -0)
📝 pyproject.toml (+2 -0)
📝 pyrightconfig.json (+1 -1)
➕ scripts/api_schema/__init__.py (+9 -0)
➕ scripts/api_schema/__main__.py (+133 -0)
➕ scripts/api_schema/live_api.py (+127 -0)
➕ scripts/api_schema/observed_shapes.py (+151 -0)
➕ scripts/api_schema/openapi_document.py (+222 -0)
➕ scripts/api_schema/unread_keys.py (+58 -0)
➕ test/unit/api_schema/observed_shapes_test.py (+136 -0)
➕ test/unit/api_schema/openapi_document_test.py (+150 -0)

📄 Description

📝 Summary

Boosty has no API spec, and our models read only part of each answer. This PR adds a dev tool that writes what Boosty really sends as an OpenAPI 3.1 file, shapes and counts only, and commits the first file.

🔧 What changes

task api:schema -- <blog> writes docs/api/boosty-api.yaml from live answers: one listing page per blog plus the newest post through the single-post endpoint. The token comes from BOOSTY_TOKEN, config.yaml or .env; --anonymous sends none, --pages N reads more pages.
The file describes the two endpoints, Post, PostsPage and one Chunk* component per chunk kind.
x-presence: how often an optional key came. x-seen-values: the vocabulary of type-like fields. x-unread-by-client: keys our models ignore.
No values from posts: no blog names, ids, titles, texts or links. Only keys, types, counts and short tokens like ok_video.
docs/api/README.md explains how to regenerate the file, how to read it and its limits.
pytest and pyright see scripts/, so the dev tools there have unit tests.

💡 Why

Unit tests check the code against our own picture of the API and stay green when Boosty changes.
The canary (#202) sees only what the models read, so new keys pass unnoticed.
The repo rule is no captured data in the tree. A schema has no data.

📌 Notes

ci/skip-changelog: nothing changes for users.

🧪 Tests

25 unit tests on made-up dicts: presence, null, the chunk split, the vocabulary rules, deterministic YAML that carries no value from the sample.
Live on the test blog with the account token: 883 lines, seven chunk kinds, 31 post keys the client ignores.
Two runs give the same bytes. openapi-spec-validator passes.

⏭️ Next

Check the synthetic test post against the schema.
Diff the live schema in the canary.
An update mode that merges new observations into the file.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed
Updated docs/ if the download flow or dev process changed
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/203 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 9/22/2026 **Status:** ✅ Merged **Merged:** 9/22/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `feat/api-schema-snapshot` --- ### 📝 Commits (3) - [`fba0390`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/fba03904617fd21d18aa1429c0057343e1ee1f28) [scripts] Infer an OpenAPI document from Boosty answers - [`f6ffb0f`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/f6ffb0f06741b30fbbed3a53bd06b4e69d14ed6a) [scripts] Add task api:schema that writes docs/api/boosty-api.yaml - [`a77594c`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/a77594c3d1b14e0f651bb1c47cc3ac79d48c3984) [scripts] Name the api_schema modules and functions after what they do ### 📊 Changes **13 files changed** (+1914 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `Taskfile.yml` (+15 -0) ➕ `docs/api/README.md` (+27 -0) ➕ `docs/api/boosty-api.yaml` (+883 -0) 📝 `pyproject.toml` (+2 -0) 📝 `pyrightconfig.json` (+1 -1) ➕ `scripts/api_schema/__init__.py` (+9 -0) ➕ `scripts/api_schema/__main__.py` (+133 -0) ➕ `scripts/api_schema/live_api.py` (+127 -0) ➕ `scripts/api_schema/observed_shapes.py` (+151 -0) ➕ `scripts/api_schema/openapi_document.py` (+222 -0) ➕ `scripts/api_schema/unread_keys.py` (+58 -0) ➕ `test/unit/api_schema/observed_shapes_test.py` (+136 -0) ➕ `test/unit/api_schema/openapi_document_test.py` (+150 -0) </details> ### 📄 Description ## 📝 Summary Boosty has no API spec, and our models read only part of each answer. This PR adds a dev tool that writes what Boosty really sends as an OpenAPI 3.1 file, shapes and counts only, and commits the first file. ## 🔧 What changes - `task api:schema -- <blog>` writes `docs/api/boosty-api.yaml` from live answers: one listing page per blog plus the newest post through the single-post endpoint. The token comes from `BOOSTY_TOKEN`, `config.yaml` or `.env`; `--anonymous` sends none, `--pages N` reads more pages. - The file describes the two endpoints, `Post`, `PostsPage` and one `Chunk*` component per chunk kind. - `x-presence`: how often an optional key came. `x-seen-values`: the vocabulary of `type`-like fields. `x-unread-by-client`: keys our models ignore. - No values from posts: no blog names, ids, titles, texts or links. Only keys, types, counts and short tokens like `ok_video`. - `docs/api/README.md` explains how to regenerate the file, how to read it and its limits. - pytest and pyright see `scripts/`, so the dev tools there have unit tests. ## 💡 Why - Unit tests check the code against our own picture of the API and stay green when Boosty changes. - The canary (#202) sees only what the models read, so new keys pass unnoticed. - The repo rule is no captured data in the tree. A schema has no data. ## 📌 Notes - `ci/skip-changelog`: nothing changes for users. ## 🧪 Tests - 25 unit tests on made-up dicts: presence, null, the chunk split, the vocabulary rules, deterministic YAML that carries no value from the sample. - Live on the test blog with the account token: 883 lines, seven chunk kinds, 31 post keys the client ignores. - Two runs give the same bytes. `openapi-spec-validator` passes. ## ⏭️ Next - Check the synthetic test post against the schema. - Diff the live schema in the canary. - An `update` mode that merges new observations into the file. ## ✅ Checklist  - [x] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed - [x] Updated `docs/` if the download flow or dev process changed - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:48 +00:00