[PR #188] [MERGED] [downloader] Remember the content types that finished when a chunk fails #187

New issue

Closed

opened 2026-10-13 19:48:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:33 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/188
Author: @Glitchy-Sheep
Created: 9/13/2026
Status: ✅ Merged
Merged: 9/13/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: fix/cache-finished-content-types

📝 Commits (1)

28147ca [downloader] Cache the content types that finished when another one fails

📊 Changes

4 files changed (+246 additions, -134 deletions)

View changed files

📝 CHANGELOG.md (+1 -0)
📝 boosty_downloader/application/use_cases/download_single_post.py (+111 -49)
📝 test/e2e/download_flow_test.py (+11 -2)
📝 test/unit/use_cases/parallel_media_test.py (+123 -83)

📄 Description

📝 Summary

Audit stage 4, part 2 of 3: audit item 19.1. A post with one dead link used to be dropped whole and re-downloaded from scratch on every one of the 5 attempts and on the next run: a post with a 2 GB video and one broken attachment transferred that video five times.

🔧 What changes

Every chunk of a post now runs to its end (gather with return_exceptions); failures are collected instead of stopping the siblings. The retry starts only after all chunks finished, so there is no race to cancel against.
Content types without a failed chunk are cached right away. The retry, and the next run, fetch only the failed type: the cache answers "files missing", not "everything missing".
post.html is rendered when every chunk that belongs on the page made it. A failed attachment does not block it (attachments are not on the page); a failed image, video or audio does, and the page waits for the retry.
Run statistics count media per finished content type, so a retry of the failed type never counts the other types twice.
Cancellation inside a chunk and unexpected errors still stop the post as a whole; containment stays in the retrier.

🧪 Tests

parallel_media_test: a dead file lets the siblings finish, the page renders, post_content is cached and files is not; a dead image blocks the page while files is cached; concurrency cap, order and Ctrl+C contract unchanged.
e2e download_flow_test with one 404: the first attempt asks for image, file, video, audio, the second for the file only (5 media requests in total), the tree and the statistics are the same as a clean run.
task ci green locally: ruff, pyright strict, import-linter, 255 tests + e2e.
Live on a small blog with three attachments the CDN answers 404 for: run 1 fails that post after 5 attempts but reports 11 images and 18 of 21 files downloaded; --dry-run afterwards promises files 3 and no images; run 2 serves 9 posts from the cache and downloads 0 media while the same post fails again on its files. failed_downloads.log holds exactly the three files.

⏭️ Next

PR 3: --post-url through the post retrier (retries, expired-link refresh, containment), URL parsing up front with a mismatch check, 429 retries at the transport.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed
Updated docs/ if the download flow or dev process changed
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/188 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 9/13/2026 **Status:** ✅ Merged **Merged:** 9/13/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `fix/cache-finished-content-types` --- ### 📝 Commits (1) - [`28147ca`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/28147ca99471172fdabf36f0bb43d00d97cae06f) [downloader] Cache the content types that finished when another one fails ### 📊 Changes **4 files changed** (+246 additions, -134 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -0) 📝 `boosty_downloader/application/use_cases/download_single_post.py` (+111 -49) 📝 `test/e2e/download_flow_test.py` (+11 -2) 📝 `test/unit/use_cases/parallel_media_test.py` (+123 -83) </details> ### 📄 Description ## 📝 Summary Audit stage 4, part 2 of 3: audit item 19.1. A post with one dead link used to be dropped whole and re-downloaded from scratch on every one of the 5 attempts and on the next run: a post with a 2 GB video and one broken attachment transferred that video five times. ## 🔧 What changes - Every chunk of a post now runs to its end (`gather` with `return_exceptions`); failures are collected instead of stopping the siblings. The retry starts only after all chunks finished, so there is no race to cancel against. - Content types without a failed chunk are cached right away. The retry, and the next run, fetch only the failed type: the cache answers "files missing", not "everything missing". - `post.html` is rendered when every chunk that belongs on the page made it. A failed attachment does not block it (attachments are not on the page); a failed image, video or audio does, and the page waits for the retry. - Run statistics count media per finished content type, so a retry of the failed type never counts the other types twice. - Cancellation inside a chunk and unexpected errors still stop the post as a whole; containment stays in the retrier. ## 🧪 Tests - `parallel_media_test`: a dead file lets the siblings finish, the page renders, `post_content` is cached and `files` is not; a dead image blocks the page while `files` is cached; concurrency cap, order and Ctrl+C contract unchanged. - e2e `download_flow_test` with one 404: the first attempt asks for image, file, video, audio, the second for the file only (5 media requests in total), the tree and the statistics are the same as a clean run. - `task ci` green locally: ruff, pyright strict, import-linter, 255 tests + e2e. - Live on a small blog with three attachments the CDN answers 404 for: run 1 fails that post after 5 attempts but reports 11 images and 18 of 21 files downloaded; `--dry-run` afterwards promises `files 3` and no images; run 2 serves 9 posts from the cache and downloads 0 media while the same post fails again on its files. `failed_downloads.log` holds exactly the three files. ## ⏭️ Next PR 3: `--post-url` through the post retrier (retries, expired-link refresh, containment), URL parsing up front with a mismatch check, 429 retries at the transport. ## ✅ Checklist - [x] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed - [x] Updated `docs/` if the download flow or dev process changed - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:33 +00:00