[PR #179] [MERGED] [downloader] Blog overview in check, download --dry-run and run statistics #179

New issue

Closed

opened 2026-10-13 19:48:24 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:48:24 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/179
Author: @Glitchy-Sheep
Created: 9/13/2026
Status: ✅ Merged
Merged: 9/13/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: feat/blog-overview

📝 Commits (2)

b0d7107 feat: Add check overview and download --dry-run
b7c44ee feat: Print run statistics at the end of a download

📊 Changes

42 files changed (+2059 additions, -120 deletions)

View changed files

📝 CHANGELOG.md (+6 -0)
📝 README.md (+3 -1)
➕ boosty_downloader/application/blog_overview.py (+166 -0)
📝 boosty_downloader/application/di/download_context.py (+4 -1)
➕ boosty_downloader/application/download_plan.py (+150 -0)
📝 boosty_downloader/application/filtering.py (+39 -0)
📝 boosty_downloader/application/mappers/audio.py (+1 -0)
📝 boosty_downloader/application/mappers/file.py (+1 -0)
📝 boosty_downloader/application/mappers/image.py (+1 -0)
➕ boosty_downloader/application/run_statistics.py (+33 -0)
📝 boosty_downloader/application/use_cases/check_total_posts.py (+23 -70)
📝 boosty_downloader/application/use_cases/download_all_posts.py (+4 -1)
📝 boosty_downloader/application/use_cases/download_single_post.py (+57 -19)
➕ boosty_downloader/application/use_cases/listing_walk.py (+73 -0)
➕ boosty_downloader/application/use_cases/plan_download.py (+77 -0)
➕ boosty_downloader/cli/blog_overview_rendering.py (+115 -0)
📝 boosty_downloader/cli/cli_options.py (+10 -0)
📝 boosty_downloader/cli/commands/check.py (+14 -4)
📝 boosty_downloader/cli/commands/download.py (+68 -7)
➕ boosty_downloader/cli/download_plan_rendering.py (+54 -0)

...and 22 more files

📄 Description

📝 Summary

Three things the tool now tells you that it kept to itself before: what a blog holds (check), what a download run would fetch before you start it (download --dry-run), and what a run actually did once it ends (a statistics block).

🔧 What changes

check prints an overview instead of three numbers: posts grouped by how they unlock (free, each subscription tier with its price, posts sold one by one), media counts by type, the posting date range and the titles this account cannot open.

Blog overview: 11 posts, 11 accessible to you

Access:
  Free: 1 post
  Tester (10 RUB/mo): 2 posts
  Single purchase (100 RUB): 6 posts
  Single purchase (300 RUB): 2 posts

Media in accessible posts:
  🖼 images: 11
  📄 files: 21
  🎬 boosty videos: 8
  🔗 external videos: 0
  🎵 audio: 2

Posts from 2026-01-04 to 2026-08-21 (last post 3 days ago)

download --dry-run shows the same overview plus the plan: new, updated and already complete posts, media to fetch by type and the known size. Nothing is downloaded and the cache is not touched.

Download plan (filters: post_content, files, audio):
  New posts: 11
  Updated or partially downloaded: 0
  Already complete: 0

Media to download:
  🖼 images: 11
  📄 files: 21
  🎬 boosty videos: 0
  🔗 external videos: 0
  🎵 audio: 2

Known download size: 28.96 MB

Every download run ends with statistics - also after a systemic stop or Ctrl+C:

Run finished in 47s
Posts: 10 downloaded, 1 failed
Media downloaded:
  🖼 images: 3
  📄 files: 18
  🎬 boosty videos: 0
  🔗 external videos: 0
  🎵 audio: 2
Total downloaded: 10.38 MB

A rerun over a cached blog stays short: Run finished in 41s: nothing new to download.

🧩 How it works

PostDTO reads subscriptionLevel, price, currencyPrices and media sizes. All tolerant: a missing or malformed value defaults instead of failing the post, because this DTO is shared with the download path. Prices are grouped by the RUB value from currencyPrices: the bare price comes in the account's display currency, so two accounts would see the same tier priced differently.
summarize_posts builds the overview from the listing alone, no extra API calls. Rendering lives in the CLI; author text is markup-escaped, so a title like [запись] cannot break or crash the output.
The dry-run plan repeats the downloader's own decisions: domain mapping (unfinished uploads and stream-only videos are not promised), the cache's missing-parts answer and the content filters. The chunk-to-filter table moved to filtering.py so both share one definition. Metadata walks list 100 posts per page (MAX_POSTS_PER_PAGE): they touch no signed media links.
Run statistics live in DownloadContext. Media is counted once per post: only the attempt that finishes lands in the totals, so a clean run matches its dry-run plan.
Found on the way: when one chunk of a post failed, gather left the sibling downloads running. They raced the retry of the same post and died at interpreter exit with Task exception was never retrieved. Now the siblings are cancelled and awaited before the error surfaces.

🧪 Tests

Unit: overview grouping and order, media counts, date range, DTO tolerance (null price, malformed tier, malformed currency map), rendering goldens for the overview, the plan and the statistics, markup escape through a real Rich console, local-calendar "days ago", the download plan against a real SQLite cache (new, complete, updated, partial, filters, unfinished uploads, locked posts), sibling cancellation.
e2e against the local Boosty-shaped server: a dry-run makes zero media requests and collapses to "everything complete" after a real run; statistics match what landed on disk; a post retried after a 404 is counted once.
Live on a real blog: check, --dry-run, a full run and a rerun. The dry-run plan minus the one post that fails on the platform side equals the run statistics; no tracebacks at exit.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed
Updated docs/ if the download flow or dev process changed
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/179 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 9/13/2026 **Status:** ✅ Merged **Merged:** 9/13/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `feat/blog-overview` --- ### 📝 Commits (2) - [`b0d7107`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/b0d7107b39034adaf0da3ec4190d60a5b58c6426) feat: Add check overview and download --dry-run - [`b7c44ee`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/b7c44ee054d4ddb64887de93e5871d363b553da8) feat: Print run statistics at the end of a download ### 📊 Changes **42 files changed** (+2059 additions, -120 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+6 -0) 📝 `README.md` (+3 -1) ➕ `boosty_downloader/application/blog_overview.py` (+166 -0) 📝 `boosty_downloader/application/di/download_context.py` (+4 -1) ➕ `boosty_downloader/application/download_plan.py` (+150 -0) 📝 `boosty_downloader/application/filtering.py` (+39 -0) 📝 `boosty_downloader/application/mappers/audio.py` (+1 -0) 📝 `boosty_downloader/application/mappers/file.py` (+1 -0) 📝 `boosty_downloader/application/mappers/image.py` (+1 -0) ➕ `boosty_downloader/application/run_statistics.py` (+33 -0) 📝 `boosty_downloader/application/use_cases/check_total_posts.py` (+23 -70) 📝 `boosty_downloader/application/use_cases/download_all_posts.py` (+4 -1) 📝 `boosty_downloader/application/use_cases/download_single_post.py` (+57 -19) ➕ `boosty_downloader/application/use_cases/listing_walk.py` (+73 -0) ➕ `boosty_downloader/application/use_cases/plan_download.py` (+77 -0) ➕ `boosty_downloader/cli/blog_overview_rendering.py` (+115 -0) 📝 `boosty_downloader/cli/cli_options.py` (+10 -0) 📝 `boosty_downloader/cli/commands/check.py` (+14 -4) 📝 `boosty_downloader/cli/commands/download.py` (+68 -7) ➕ `boosty_downloader/cli/download_plan_rendering.py` (+54 -0) _...and 22 more files_ </details> ### 📄 Description ## 📝 Summary Three things the tool now tells you that it kept to itself before: what a blog holds (`check`), what a download run would fetch before you start it (`download --dry-run`), and what a run actually did once it ends (a statistics block). ## 🔧 What changes **`check`** prints an overview instead of three numbers: posts grouped by how they unlock (free, each subscription tier with its price, posts sold one by one), media counts by type, the posting date range and the titles this account cannot open. ``` Blog overview: 11 posts, 11 accessible to you Access: Free: 1 post Tester (10 RUB/mo): 2 posts Single purchase (100 RUB): 6 posts Single purchase (300 RUB): 2 posts Media in accessible posts: 🖼 images: 11 📄 files: 21 🎬 boosty videos: 8 🔗 external videos: 0 🎵 audio: 2 Posts from 2026-01-04 to 2026-08-21 (last post 3 days ago) ``` **`download --dry-run`** shows the same overview plus the plan: new, updated and already complete posts, media to fetch by type and the known size. Nothing is downloaded and the cache is not touched. ``` Download plan (filters: post_content, files, audio): New posts: 11 Updated or partially downloaded: 0 Already complete: 0 Media to download: 🖼 images: 11 📄 files: 21 🎬 boosty videos: 0 🔗 external videos: 0 🎵 audio: 2 Known download size: 28.96 MB ``` **Every `download` run** ends with statistics - also after a systemic stop or Ctrl+C: ``` Run finished in 47s Posts: 10 downloaded, 1 failed Media downloaded: 🖼 images: 3 📄 files: 18 🎬 boosty videos: 0 🔗 external videos: 0 🎵 audio: 2 Total downloaded: 10.38 MB ``` A rerun over a cached blog stays short: `Run finished in 41s: nothing new to download`. ## 🧩 How it works 1. `PostDTO` reads `subscriptionLevel`, `price`, `currencyPrices` and media sizes. All tolerant: a missing or malformed value defaults instead of failing the post, because this DTO is shared with the download path. Prices are grouped by the RUB value from `currencyPrices`: the bare `price` comes in the account's display currency, so two accounts would see the same tier priced differently. 2. `summarize_posts` builds the overview from the listing alone, no extra API calls. Rendering lives in the CLI; author text is markup-escaped, so a title like `[запись]` cannot break or crash the output. 3. The dry-run plan repeats the downloader's own decisions: domain mapping (unfinished uploads and stream-only videos are not promised), the cache's missing-parts answer and the content filters. The chunk-to-filter table moved to `filtering.py` so both share one definition. Metadata walks list 100 posts per page (`MAX_POSTS_PER_PAGE`): they touch no signed media links. 4. Run statistics live in `DownloadContext`. Media is counted once per post: only the attempt that finishes lands in the totals, so a clean run matches its dry-run plan. 5. Found on the way: when one chunk of a post failed, `gather` left the sibling downloads running. They raced the retry of the same post and died at interpreter exit with `Task exception was never retrieved`. Now the siblings are cancelled and awaited before the error surfaces. ## 🧪 Tests - Unit: overview grouping and order, media counts, date range, DTO tolerance (null price, malformed tier, malformed currency map), rendering goldens for the overview, the plan and the statistics, markup escape through a real Rich console, local-calendar "days ago", the download plan against a real SQLite cache (new, complete, updated, partial, filters, unfinished uploads, locked posts), sibling cancellation. - e2e against the local Boosty-shaped server: a dry-run makes zero media requests and collapses to "everything complete" after a real run; statistics match what landed on disk; a post retried after a 404 is counted once. - Live on a real blog: `check`, `--dry-run`, a full run and a rerun. The dry-run plan minus the one post that fails on the platform side equals the run statistics; no tracebacks at exit. ## ✅ Checklist - [x] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed - [x] Updated `docs/` if the download flow or dev process changed - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:48:24 +00:00