[PR #170] [MERGED] [downloader] Download post media in parallel #171

New issue

Closed

opened 2026-10-13 19:48:18 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:18 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/170
Author: @Glitchy-Sheep
Created: 8/22/2026
Status: ✅ Merged
Merged: 8/23/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: feat/parallel-media

📝 Commits (3)

4280033 fix: Run yt-dlp off the event loop
a43f996 feat: Download post media in parallel
12042a9 test: Prove parallel downloads and changelog the speedup

📊 Changes

6 files changed (+449 additions, -15 deletions)

View changed files

📝 CHANGELOG.md (+5 -0)
📝 boosty_downloader/application/use_cases/download_single_post.py (+62 -13)
📝 boosty_downloader/infrastructure/file_downloader.py (+3 -1)
📝 test/e2e/download_flow_test.py (+35 -1)
➕ test/unit/use_cases/ext_video_threading_test.py (+138 -0)
➕ test/unit/use_cases/parallel_media_test.py (+206 -0)

📄 Description

[downloader] Download post media in parallel

Summary

Media of one post now downloads up to 4 files at once instead of strictly one by one. Measured on live CDN before designing: the CDN caps every single connection (~8-15 MB/s cold) and a cold small-file request pays seconds of latency, so a wide channel sat idle. Parallelism reclaims it: ~10x on image-heavy posts, 2-3x on video posts.

Live measurements that drove the design

Scenario (cold CDN, each file fetched once)	Sequential	Parallel
4 images x 2.4 MB	67s	6.5s (4 conns)
Big video variants	8-15 MB/s	17 MB/s (3) / 34 MB/s (6)

Four connections keep most of the win without hammering the host; posts still go one by one (deliberate - progress readability, the failure-streak breaker and cache semantics stay intact).

Changes

yt-dlp moves off the event loop (asyncio.to_thread): an external video no longer freezes every other download and the progress display. Progress updates are marshalled onto the loop thread before touching Rich; Ctrl+C still aborts the download - the cancel flag raises KeyboardInterrupt inside the worker, yt-dlp's own abort path
_process_chunks_concurrently - a named private step of the use case: semaphore(4) + gather. Two invariants are pinned: the page keeps the author's chunk order whatever finishes first, and the error contract is unchanged (a failed chunk surfaces as its own error, an outer cancel maps to the application cancellation that the 130 exit code relies on)
Download chunk size grows 512 KiB -> 2 MiB: less per-chunk bookkeeping on fast channels

Verification

task ci green: 182 tests + 1 xfail (7 new)
New tests: thread/loop separation and cancellation abort for yt-dlp; overlap + cap + order + error semantics through the real execute(); a slow-server e2e where four 0.4s media must beat the sequential lower bound
Live: a real paid post (image + 9 files) downloads in 3.8s end-to-end; a single cold image alone used to cost ~17s
Changelog: two entries under Added

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/170 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 8/22/2026 **Status:** ✅ Merged **Merged:** 8/23/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `feat/parallel-media` --- ### 📝 Commits (3) - [`4280033`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/4280033f89dd6da0cbf62666a3a707c7be827dd4) fix: Run yt-dlp off the event loop - [`a43f996`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/a43f996afec7605c7d253d0a1f1ed7d123842e99) feat: Download post media in parallel - [`12042a9`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/12042a94321284a83afbc32854f341901c602493) test: Prove parallel downloads and changelog the speedup ### 📊 Changes **6 files changed** (+449 additions, -15 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+5 -0) 📝 `boosty_downloader/application/use_cases/download_single_post.py` (+62 -13) 📝 `boosty_downloader/infrastructure/file_downloader.py` (+3 -1) 📝 `test/e2e/download_flow_test.py` (+35 -1) ➕ `test/unit/use_cases/ext_video_threading_test.py` (+138 -0) ➕ `test/unit/use_cases/parallel_media_test.py` (+206 -0) </details> ### 📄 Description # [downloader] Download post media in parallel ## Summary Media of one post now downloads up to 4 files at once instead of strictly one by one. Measured on live CDN before designing: the CDN caps every single connection (~8-15 MB/s cold) and a cold small-file request pays seconds of latency, so a wide channel sat idle. Parallelism reclaims it: ~10x on image-heavy posts, 2-3x on video posts. ## Live measurements that drove the design | Scenario (cold CDN, each file fetched once) | Sequential | Parallel | |---|---|---| | 4 images x 2.4 MB | 67s | 6.5s (4 conns) | | Big video variants | 8-15 MB/s | 17 MB/s (3) / 34 MB/s (6) | Four connections keep most of the win without hammering the host; posts still go one by one (deliberate - progress readability, the failure-streak breaker and cache semantics stay intact). ## Changes 1. **yt-dlp moves off the event loop** (`asyncio.to_thread`): an external video no longer freezes every other download and the progress display. Progress updates are marshalled onto the loop thread before touching Rich; Ctrl+C still aborts the download - the cancel flag raises `KeyboardInterrupt` inside the worker, yt-dlp's own abort path 2. **`_process_chunks_concurrently`** - a named private step of the use case: semaphore(4) + `gather`. Two invariants are pinned: the page keeps the author's chunk order whatever finishes first, and the error contract is unchanged (a failed chunk surfaces as its own error, an outer cancel maps to the application cancellation that the 130 exit code relies on) 3. Download chunk size grows 512 KiB -> 2 MiB: less per-chunk bookkeeping on fast channels ## Verification - `task ci` green: **182 tests + 1 xfail** (7 new) - New tests: thread/loop separation and cancellation abort for yt-dlp; overlap + cap + order + error semantics through the real `execute()`; a slow-server e2e where four 0.4s media must beat the sequential lower bound - Live: a real paid post (image + 9 files) downloads in 3.8s end-to-end; a single cold image alone used to cost ~17s - Changelog: two entries under Added --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:18 +00:00