[PR #197] [MERGED] [downloader] Skip a post with a gone external video without retries #196

New issue

Closed

opened 2026-10-13 19:48:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:40 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/197
Author: @Glitchy-Sheep
Created: 9/20/2026
Status: ✅ Merged
Merged: 9/21/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: fix/gone-external-video

📝 Commits (3)

95ae938 [ext-video] Tell a gone video from a network failure
74e0bb2 [downloader] Skip a post with a gone external video without retries
6b769ae [docs] Describe how a gone external video is skipped

📊 Changes

17 files changed (+429 additions, -17 deletions)

View changed files

📝 CHANGELOG.md (+1 -0)
📝 README.md (+1 -0)
📝 boosty_downloader/application/exceptions/application_errors.py (+6 -1)
📝 boosty_downloader/application/post_retry.py (+20 -0)
📝 boosty_downloader/application/use_cases/download_all_posts.py (+4 -0)
📝 boosty_downloader/application/use_cases/download_post_by_id.py (+1 -1)
📝 boosty_downloader/application/use_cases/download_single_post.py (+1 -0)
📝 boosty_downloader/cli/commands/download.py (+1 -1)
📝 boosty_downloader/infrastructure/external_videos_downloader/external_videos_downloader.py (+47 -3)
📝 boosty_downloader/infrastructure/post_media_downloader.py (+16 -1)
📝 docs/concepts/boosty/download-flow.md (+1 -0)
📝 test/e2e/download_flow_test.py (+69 -10)
➕ test/unit/external_videos_downloader/gone_video_test.py (+161 -0)
📝 test/unit/post_media_downloader/post_media_downloader_test.py (+25 -0)
📝 test/unit/use_cases/chunk_error_translation_test.py (+15 -0)
📝 test/unit/use_cases/download_all_posts_test.py (+31 -0)
📝 test/unit/use_cases/download_post_by_id_test.py (+29 -0)

📄 Description

📝 Summary

Closes #76. A YouTube or Vimeo video that is gone for good no longer costs five attempts.

🔧 What changes

A deleted, private, blocked or unsupported external video ends the post after the first attempt. The console says Skip post, retrying will not help: <post> (External video unavailable: This video is unavailable); the reason is the site's own answer.
The rest of the post is saved as before, the page marks the video's place with the reason and a link to the original (#196), and failed_downloads.log gets the entry.
The post counts as failed, but not toward the 5-failures-in-a-row stop: a deleted video says nothing about the disk or the network. --post-url still exits with 1.
Network failures, 5xx, 429 and expired links keep their retries and the link refresh.

How the two are told apart

yt-dlp keeps the original exception in DownloadError.exc_info. An ExtractorError marked expected is the site's normal answer (deleted, private, geo-blocked, unsupported url, "sign in to confirm"); a 404 or 410 of the video page means the same. Network failures come as TransportError and never carry the mark. The wrapper raises ExtVideoUnavailableError(url, reason) for the first group; MediaDownloadError and ApplicationFailedDownloadError carry a retryable flag to the retrier, which returns a new PostOutcome.unavailable.

🚫 Not changed on purpose

No new config knobs for attempts or timeouts: retrying the impossible was the problem, not the numbers. yt-dlp keeps its own socket timeout and fragment retries.
The post stays failed and post_content stays out of the cache, so the page with the placeholder is rewritten on every run and never freezes. The cost is one probe of the video per run (about 2 seconds).

🧪 Tests

yt-dlp wrapper: 3 gone cases with the reason and the url, 3 transient cases, and the download phase; yt-dlp is faked at the module seam with its real error types.
Media downloader and use case: the flag and the message travel with the error.
Retrier: --post-url with a gone video ends after one attempt; a full run with six gone videos in a row does not stop and gives every post one attempt.
e2e: the fixture post plus a dead link with a yt-dlp stand-in - 4 media requests, a failed post, the placeholder with the reason and the original link on the page, the log entry.
task ci green locally: ruff, pyright strict, import-linter, 307 tests + e2e.
Stand with the real yt-dlp against a deleted YouTube id through the whole pipeline: the first run takes 4 seconds and one attempt, the second run 2 seconds, the page and the log look as above.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed
Updated docs/ if the download flow or dev process changed
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/197 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 9/20/2026 **Status:** ✅ Merged **Merged:** 9/21/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `fix/gone-external-video` --- ### 📝 Commits (3) - [`95ae938`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/95ae938046bab1921bf1e57891fe3d6f7580a83c) [ext-video] Tell a gone video from a network failure - [`74e0bb2`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/74e0bb2921d5431e770480dcf7239d84822f2c9e) [downloader] Skip a post with a gone external video without retries - [`6b769ae`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/6b769aebc172b8389572f08fc1174fd740224ea4) [docs] Describe how a gone external video is skipped ### 📊 Changes **17 files changed** (+429 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -0) 📝 `README.md` (+1 -0) 📝 `boosty_downloader/application/exceptions/application_errors.py` (+6 -1) 📝 `boosty_downloader/application/post_retry.py` (+20 -0) 📝 `boosty_downloader/application/use_cases/download_all_posts.py` (+4 -0) 📝 `boosty_downloader/application/use_cases/download_post_by_id.py` (+1 -1) 📝 `boosty_downloader/application/use_cases/download_single_post.py` (+1 -0) 📝 `boosty_downloader/cli/commands/download.py` (+1 -1) 📝 `boosty_downloader/infrastructure/external_videos_downloader/external_videos_downloader.py` (+47 -3) 📝 `boosty_downloader/infrastructure/post_media_downloader.py` (+16 -1) 📝 `docs/concepts/boosty/download-flow.md` (+1 -0) 📝 `test/e2e/download_flow_test.py` (+69 -10) ➕ `test/unit/external_videos_downloader/gone_video_test.py` (+161 -0) 📝 `test/unit/post_media_downloader/post_media_downloader_test.py` (+25 -0) 📝 `test/unit/use_cases/chunk_error_translation_test.py` (+15 -0) 📝 `test/unit/use_cases/download_all_posts_test.py` (+31 -0) 📝 `test/unit/use_cases/download_post_by_id_test.py` (+29 -0) </details> ### 📄 Description ## 📝 Summary Closes #76. A YouTube or Vimeo video that is gone for good no longer costs five attempts. ## 🔧 What changes - A deleted, private, blocked or unsupported external video ends the post after the first attempt. The console says `Skip post, retrying will not help: <post> (External video unavailable: This video is unavailable)`; the reason is the site's own answer. - The rest of the post is saved as before, the page marks the video's place with the reason and a link to the original (#196), and `failed_downloads.log` gets the entry. - The post counts as failed, but not toward the 5-failures-in-a-row stop: a deleted video says nothing about the disk or the network. `--post-url` still exits with 1. - Network failures, 5xx, 429 and expired links keep their retries and the link refresh. ## How the two are told apart yt-dlp keeps the original exception in `DownloadError.exc_info`. An `ExtractorError` marked `expected` is the site's normal answer (deleted, private, geo-blocked, unsupported url, "sign in to confirm"); a 404 or 410 of the video page means the same. Network failures come as `TransportError` and never carry the mark. The wrapper raises `ExtVideoUnavailableError(url, reason)` for the first group; `MediaDownloadError` and `ApplicationFailedDownloadError` carry a `retryable` flag to the retrier, which returns a new `PostOutcome.unavailable`. ## 🚫 Not changed on purpose - No new config knobs for attempts or timeouts: retrying the impossible was the problem, not the numbers. yt-dlp keeps its own socket timeout and fragment retries. - The post stays failed and `post_content` stays out of the cache, so the page with the placeholder is rewritten on every run and never freezes. The cost is one probe of the video per run (about 2 seconds). ## 🧪 Tests - yt-dlp wrapper: 3 gone cases with the reason and the url, 3 transient cases, and the download phase; yt-dlp is faked at the module seam with its real error types. - Media downloader and use case: the flag and the message travel with the error. - Retrier: `--post-url` with a gone video ends after one attempt; a full run with six gone videos in a row does not stop and gives every post one attempt. - e2e: the fixture post plus a dead link with a yt-dlp stand-in - 4 media requests, a failed post, the placeholder with the reason and the original link on the page, the log entry. - `task ci` green locally: ruff, pyright strict, import-linter, 307 tests + e2e. - Stand with the real yt-dlp against a deleted YouTube id through the whole pipeline: the first run takes 4 seconds and one attempt, the second run 2 seconds, the page and the log look as above. ## ✅ Checklist - [x] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed - [x] Updated `docs/` if the download flow or dev process changed - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:40 +00:00