[PR #189] [MERGED] [downloader] Give --post-url the same retries and refresh as the full run #188

New issue

Closed

opened 2026-10-13 19:48:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/189
Author: @Glitchy-Sheep
Created: 9/14/2026
Status: ✅ Merged
Merged: 9/14/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: fix/post-url-through-retrier

📝 Commits (4)

d6e27e4 [cli] Parse --post-url up front and reject a post of another creator
c8d8dd5 [downloader] Run --post-url through the post retrier
1935f98 [downloader] Retry 429 answers and describe the download flow as it is
a2abe2d [tests] Read CLI error text without terminal colors in the smoke tests

📊 Changes

11 files changed (+399 additions, -223 deletions)

View changed files

📝 CHANGELOG.md (+7 -0)
📝 README.md (+5 -2)
➕ boosty_downloader/application/use_cases/download_post_by_id.py (+115 -0)
➖ boosty_downloader/application/use_cases/download_specific_post.py (+0 -151)
📝 boosty_downloader/cli/commands/download.py (+51 -29)
📝 boosty_downloader/cli/composition_root.py (+7 -0)
➕ boosty_downloader/infrastructure/boosty_api/utils/post_url.py (+41 -0)
📝 docs/concepts/boosty/download-flow.md (+25 -15)
➕ test/unit/boosty_api/post_url_test.py (+47 -0)
📝 test/unit/cli/cli_smoke_test.py (+25 -2)
📝 test/unit/use_cases/download_post_by_id_test.py (+76 -24)

📄 Description

📝 Summary

Audit stage 4, part 3 of 3: --post-url gets the protection the full run already had, the post link is checked before anything runs, and the transport retries 429.

🔧 What changes

Fixed

download --post-url went around the post retrier: one attempt, no refresh of expired signed links, no containment of unexpected errors, no statistics block. It now runs through PostDownloadRetrier like every post of a full run: 5 attempts with growing pauses, one re-fetch of the post when a link dies mid-way, a skip with a clear message on an unexpected error. A failed download still exits with 1.
download A --post-url https://boosty.to/B/posts/<id> asked the API for B's post and saved it under A's folder, cache and failure-log URL. The link is now parsed up front with yarl: a post of another creator is a usage error with a hint to run the command with that creator's name, and a link that is not a post page is rejected with the expected shape. Nothing is read or requested before that check.
The use case is DownloadPostByIdUseCase: the command hands it the id, and the API call, the folder, the cache and the log all use the same creator.

Changed

HTTP 429 answers are retried like connection errors and 5xx, and the pauses between transport retries are 2, 4, 8 and 16 seconds instead of a 0.2 to 1.6 second ladder. Body failures and expired links remain the post retrier's job; the comment next to the options says which level owns what.

📚 Docs

docs/concepts/boosty/download-flow.md shows the retrier and PostMediaDownloader in the pipeline, has a "When a piece fails" section, and no longer claims that expired links end the run.
README "When downloads fail" mentions the retries, the per-type resume and --post-url.

🧪 Tests

post_url_test: four link shapes users paste (plain, trailing slash, query, www plus fragment) and six rejections.
CLI smoke: a non-post link and another creator's post fail with exit 2 before the config is read.
download_post_by_id_test: a dead link is retried and becomes a failed outcome with posts_failed counted; a 400 mid-way fetches the post again and the download succeeds; a locked post counts as posts_locked; cancellation still propagates.
task ci green locally: ruff, pyright strict, import-linter, 268 tests + e2e.
Live on a small blog: a post of another creator and a blog-page link are rejected with the messages above; --post-url on a post with three dead attachments shows the attempts, the skip, the summary and the statistics block (8 images saved on the first attempt), exits with 1, and a rerun downloads nothing new.

This closes audit stage 4.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed
Updated docs/ if the download flow or dev process changed
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/189 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 9/14/2026 **Status:** ✅ Merged **Merged:** 9/14/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `fix/post-url-through-retrier` --- ### 📝 Commits (4) - [`d6e27e4`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/d6e27e436d0c3e256ded77440c86c944e79bdbfd) [cli] Parse --post-url up front and reject a post of another creator - [`c8d8dd5`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/c8d8dd54125aa5f5f85e061a71e4dcd8deee89fa) [downloader] Run --post-url through the post retrier - [`1935f98`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/1935f98171de41010a8e4d39bbda2ba89b7beb1d) [downloader] Retry 429 answers and describe the download flow as it is - [`a2abe2d`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/a2abe2d58f435fa7538dc84acd3cda1e1ffbeb3d) [tests] Read CLI error text without terminal colors in the smoke tests ### 📊 Changes **11 files changed** (+399 additions, -223 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+7 -0) 📝 `README.md` (+5 -2) ➕ `boosty_downloader/application/use_cases/download_post_by_id.py` (+115 -0) ➖ `boosty_downloader/application/use_cases/download_specific_post.py` (+0 -151) 📝 `boosty_downloader/cli/commands/download.py` (+51 -29) 📝 `boosty_downloader/cli/composition_root.py` (+7 -0) ➕ `boosty_downloader/infrastructure/boosty_api/utils/post_url.py` (+41 -0) 📝 `docs/concepts/boosty/download-flow.md` (+25 -15) ➕ `test/unit/boosty_api/post_url_test.py` (+47 -0) 📝 `test/unit/cli/cli_smoke_test.py` (+25 -2) 📝 `test/unit/use_cases/download_post_by_id_test.py` (+76 -24) </details> ### 📄 Description ## 📝 Summary Audit stage 4, part 3 of 3: `--post-url` gets the protection the full run already had, the post link is checked before anything runs, and the transport retries 429. ## 🔧 What changes **Fixed** - `download --post-url` went around the post retrier: one attempt, no refresh of expired signed links, no containment of unexpected errors, no statistics block. It now runs through `PostDownloadRetrier` like every post of a full run: 5 attempts with growing pauses, one re-fetch of the post when a link dies mid-way, a skip with a clear message on an unexpected error. A failed download still exits with 1. - `download A --post-url https://boosty.to/B/posts/<id>` asked the API for B's post and saved it under A's folder, cache and failure-log URL. The link is now parsed up front with yarl: a post of another creator is a usage error with a hint to run the command with that creator's name, and a link that is not a post page is rejected with the expected shape. Nothing is read or requested before that check. - The use case is `DownloadPostByIdUseCase`: the command hands it the id, and the API call, the folder, the cache and the log all use the same creator. **Changed** - HTTP 429 answers are retried like connection errors and 5xx, and the pauses between transport retries are 2, 4, 8 and 16 seconds instead of a 0.2 to 1.6 second ladder. Body failures and expired links remain the post retrier's job; the comment next to the options says which level owns what. ## 📚 Docs - `docs/concepts/boosty/download-flow.md` shows the retrier and `PostMediaDownloader` in the pipeline, has a "When a piece fails" section, and no longer claims that expired links end the run. - README "When downloads fail" mentions the retries, the per-type resume and `--post-url`. ## 🧪 Tests - `post_url_test`: four link shapes users paste (plain, trailing slash, query, www plus fragment) and six rejections. - CLI smoke: a non-post link and another creator's post fail with exit 2 before the config is read. - `download_post_by_id_test`: a dead link is retried and becomes a failed outcome with `posts_failed` counted; a 400 mid-way fetches the post again and the download succeeds; a locked post counts as `posts_locked`; cancellation still propagates. - `task ci` green locally: ruff, pyright strict, import-linter, 268 tests + e2e. - Live on a small blog: a post of another creator and a blog-page link are rejected with the messages above; `--post-url` on a post with three dead attachments shows the attempts, the skip, the summary and the statistics block (8 images saved on the first attempt), exits with 1, and a rerun downloads nothing new. This closes audit stage 4. ## ✅ Checklist - [x] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed - [x] Updated `docs/` if the download flow or dev process changed - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:34 +00:00