[PR #150] [MERGED] [downloader] Re-fetch posts with expired signed links and fetch single posts directly #154

New issue

Closed

opened 2026-10-13 19:48:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:03 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/150
Author: @Glitchy-Sheep
Created: 8/15/2026
Status: ✅ Merged
Merged: 8/15/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: feat/single-post-refetch

📝 Commits (4)

d670377 feat: Fetch a single post through the dedicated API endpoint
da3c3cb feat: Download a post by url with one direct API request
2126aec fix: Refresh expired signed links once instead of dying
f003a8f docs: Note the expired-link refresh in the changelog

📊 Changes

9 files changed (+661 additions, -173 deletions)

View changed files

📝 CHANGELOG.md (+5 -0)
➕ boosty_downloader/src/application/post_retry.py (+194 -0)
📝 boosty_downloader/src/application/use_cases/download_all_posts.py (+14 -126)
📝 boosty_downloader/src/application/use_cases/download_specific_post.py (+37 -44)
📝 boosty_downloader/src/infrastructure/boosty_api/core/client.py (+41 -0)
📝 boosty_downloader/src/infrastructure/file_downloader.py (+20 -0)
📝 test/unit/boosty_api/client_test.py (+41 -0)
📝 test/unit/use_cases/download_all_posts_test.py (+113 -3)
➕ test/unit/use_cases/download_specific_post_test.py (+196 -0)

📄 Description

Summary

Signed download links live about 24 hours from the moment a listing page is fetched. On long runs they died mid-download: five retries hit the same dead url, the post went to the failed log, and five such posts in a row stopped the whole run. Now a dead link triggers one re-fetch of the post - fresh signatures, immediate retry. On top of that, --post-url uses the newly discovered single-post API endpoint: one direct request instead of walking the blog page by page.

Problem

A CDN 400/403 for an expired signature was flattened into a generic status error nobody inspected
The per-post retry loop hit the same dead url five times in ~8 seconds - guaranteed failure
An expired listing page means every post on it fails: the failure streak stopped the run as "systemic"
The only cure was a manual rerun; the client had no way to ask the API for one post - --post-url linear-scanned 100-post pages

Fix

get_single_post on the API client: GET blog/{author}/post/{id}, one call, fresh signed urls; a missing post and an unparsable post raise distinct, clearly-worded errors
--post-url asks for the post directly; a paywalled post without access now says so instead of downloading an empty preview
is_expired_link_error recognizes a CDN 400/403 anywhere in the error cause chain
The retry policy moved into its own PostDownloadRetrier (next to FailureStreakBreaker): attempt budget, backoff, and one expired-link refresh per post - a healed post touches neither the failed log nor the failure streak
The refresh is spent even when it fails: a second 400 on fresh links means the problem is not expiry, and the old retry path takes over

Before / After

Before: a run longer than the signature lifetime ends with "Skip post after 5 failed attempts" for every post of a stale page, then the systemic-failure stop; the user reruns by hand.

After: the log says Signed links look expired, refreshing the post: ..., the post re-fetches and downloads; --post-url starts downloading instantly with links fetched seconds ago.

Verification

Live --debug run of --post-url: the debug log shows exactly one API request - the direct GET .../post/{id} - and the post downloads
11 new unit tests: the client endpoint (parse, 404, validation details, 401), the direct --post-url flow (download, not found, unparsable, no access), and the refresh policy (healed on second try; refresh spent once; API failure falls back to the old retry path)
Refactoring is invisible to the existing suite: all containment and failure-streak tests pass without a single expectation change; 124 tests total
Full task ci green: checks, unit tests, 6 live integration tests, build

Notes

Expiry itself cannot be reproduced live in one session (needs a 24h-old link) - the trigger and the healing are pinned by unit tests on the real error types
The listing page stays at 5 posts: the refresh is the safety net for the residual window, not a license for huge pages

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/150 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 8/15/2026 **Status:** ✅ Merged **Merged:** 8/15/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `feat/single-post-refetch` --- ### 📝 Commits (4) - [`d670377`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/d6703771e396197e8d52c422b5d344d74e17c15b) feat: Fetch a single post through the dedicated API endpoint - [`da3c3cb`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/da3c3cb836a0acac6954dbd9817d32351e2e47d0) feat: Download a post by url with one direct API request - [`2126aec`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/2126aec759592e850f23c2ec422c48b08e23c482) fix: Refresh expired signed links once instead of dying - [`f003a8f`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/f003a8f5dc088a124ec88098cc39e740daf6b311) docs: Note the expired-link refresh in the changelog ### 📊 Changes **9 files changed** (+661 additions, -173 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+5 -0) ➕ `boosty_downloader/src/application/post_retry.py` (+194 -0) 📝 `boosty_downloader/src/application/use_cases/download_all_posts.py` (+14 -126) 📝 `boosty_downloader/src/application/use_cases/download_specific_post.py` (+37 -44) 📝 `boosty_downloader/src/infrastructure/boosty_api/core/client.py` (+41 -0) 📝 `boosty_downloader/src/infrastructure/file_downloader.py` (+20 -0) 📝 `test/unit/boosty_api/client_test.py` (+41 -0) 📝 `test/unit/use_cases/download_all_posts_test.py` (+113 -3) ➕ `test/unit/use_cases/download_specific_post_test.py` (+196 -0) </details> ### 📄 Description ## Summary Signed download links live about 24 hours from the moment a listing page is fetched. On long runs they died mid-download: five retries hit the same dead url, the post went to the failed log, and five such posts in a row stopped the whole run. Now a dead link triggers one re-fetch of the post - fresh signatures, immediate retry. On top of that, `--post-url` uses the newly discovered single-post API endpoint: one direct request instead of walking the blog page by page. ## Problem - A CDN 400/403 for an expired signature was flattened into a generic status error nobody inspected - The per-post retry loop hit the same dead url five times in ~8 seconds - guaranteed failure - An expired listing page means every post on it fails: the failure streak stopped the run as "systemic" - The only cure was a manual rerun; the client had no way to ask the API for one post - `--post-url` linear-scanned 100-post pages ## Fix - `get_single_post` on the API client: `GET blog/{author}/post/{id}`, one call, fresh signed urls; a missing post and an unparsable post raise distinct, clearly-worded errors - `--post-url` asks for the post directly; a paywalled post without access now says so instead of downloading an empty preview - `is_expired_link_error` recognizes a CDN 400/403 anywhere in the error cause chain - The retry policy moved into its own `PostDownloadRetrier` (next to `FailureStreakBreaker`): attempt budget, backoff, and one expired-link refresh per post - a healed post touches neither the failed log nor the failure streak - The refresh is spent even when it fails: a second 400 on fresh links means the problem is not expiry, and the old retry path takes over ## Before / After **Before:** a run longer than the signature lifetime ends with "Skip post after 5 failed attempts" for every post of a stale page, then the systemic-failure stop; the user reruns by hand. **After:** the log says `Signed links look expired, refreshing the post: ...`, the post re-fetches and downloads; `--post-url` starts downloading instantly with links fetched seconds ago. ## Verification - Live `--debug` run of `--post-url`: the debug log shows exactly one API request - the direct `GET .../post/{id}` - and the post downloads - 11 new unit tests: the client endpoint (parse, 404, validation details, 401), the direct `--post-url` flow (download, not found, unparsable, no access), and the refresh policy (healed on second try; refresh spent once; API failure falls back to the old retry path) - Refactoring is invisible to the existing suite: all containment and failure-streak tests pass without a single expectation change; 124 tests total - Full `task ci` green: checks, unit tests, 6 live integration tests, build ## Notes - Expiry itself cannot be reproduced live in one session (needs a 24h-old link) - the trigger and the healing are pinned by unit tests on the real error types - The listing page stays at 5 posts: the refresh is the safety net for the residual window, not a license for huge pages --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:03 +00:00