[PR #222] [MERGED] feat(application): reconcile stored posts and plan missing downloads #220

New issue

Closed

opened 2026-10-13 19:49:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:49:00 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/222
Author: @Glitchy-Sheep
Created: 10/7/2026
Status: ✅ Merged
Merged: 10/7/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: feat/reconcile-stored-posts

📝 Commits (7)

6e5c28b refactor(application): keep a complete live post for reconciliation
f4a0781 feat(application): preserve media order and reserve file paths
2057626 feat(application): merge post content with retained media history
1bed5b2 feat(application): track saved media files from disk snapshots
97d17c4 refactor(application): share media names with file discovery
80e4b0b feat(application): recover media state from existing files
027c83d feat(application): reconcile posts and plan missing media downloads

📊 Changes

35 files changed (+4004 additions, -108 deletions)

View changed files

📝 CHANGELOG.md (+1 -0)
➕ boosty_downloader/application/_media_order.py (+30 -0)
📝 boosty_downloader/application/filtering.py (+11 -0)
📝 boosty_downloader/application/mappers/link_header_text.py (+4 -1)
📝 boosty_downloader/application/mappers/list.py (+12 -6)
➕ boosty_downloader/application/mappers/live_post.py (+292 -0)
➕ boosty_downloader/application/media_paths.py (+258 -0)
➕ boosty_downloader/application/reconcile.py (+289 -0)
📝 boosty_downloader/cli/views/blog_overview.py (+31 -3)
📝 boosty_downloader/domain/stored_post.py (+11 -3)
📝 boosty_downloader/infrastructure/boosty_api/models/post/post_data_types/post_data_text.py (+3 -1)
📝 boosty_downloader/infrastructure/file_downloader.py (+6 -7)
📝 boosty_downloader/infrastructure/html_generator/renderer.py (+4 -1)
📝 boosty_downloader/infrastructure/html_generator/templates/list.html (+0 -1)
➕ boosty_downloader/infrastructure/media_filenames.py (+20 -0)
📝 boosty_downloader/infrastructure/path_sanitizer.py (+25 -1)
📝 boosty_downloader/infrastructure/post_caching/post_cache.py (+4 -1)
📝 boosty_downloader/infrastructure/post_media_downloader.py (+2 -19)
📝 scripts/api_schema/__main__.py (+22 -17)
📝 scripts/api_schema/schema_changes.py (+17 -6)

...and 15 more files

📄 Description

📝 Summary

Add the core needed to save a complete post across partial downloads: text and images today, then video later. It combines the saved record, the current post and a disk snapshot to update media state and list missing downloads.

🔧 What changes

Map the full post body and media metadata, including uploads that are not ready to download.
Keep saved paths, timestamps and media removed by the author when refreshing a post.
Observe saved files and recover completed copies before planning downloads, with each path reserved for its owner.
Select ready missing media by content filter, restore option and unavailable IDs already checked during the run.
Share filename rules with the existing downloader, including portable handling of colons, backslashes and unusually long extensions.
Use named records, enums and small formatting helpers in the related CLI, HTML, cache and schema-report code.

The new mapper and reconciliation API currently have test callers only. The existing downloader already uses the shared filename helpers; dry-run and download execution will use reconciliation in later changes.

🧪 Tests

task ci passed: 693 unit and local HTTP e2e tests, Ruff, formatting, Pyright, import boundaries and package builds.
Cases cover partial runs, edited posts, removed media, restored files, name collisions, lost access, unfinished uploads and retries.
Local HTTP tests verify that discovery finds files written by the actual downloader under the shared naming rules.
Independent code and task reviews passed. The branch merges cleanly with current main.
Live Boosty integration was not run: this worktree has no .env, and the new reconciliation API has no CLI entry point yet.

⏭️ Next

Use reconciliation in download_plan.py to count exact missing media during a dry run. Later changes connect record persistence, download execution and full-page rendering.

✅ Checklist

Tested live on a real blog (feature or fix)
Updated CLI --help and README if needed (no CLI interface changes)
Updated docs/ if the download flow or dev process changed (neither changes in this PR)
Added a line to ## Unreleased in CHANGELOG.md (or the ci/skip-changelog label if not user-facing)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/222 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 10/7/2026 **Status:** ✅ Merged **Merged:** 10/7/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `feat/reconcile-stored-posts` --- ### 📝 Commits (7) - [`6e5c28b`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/6e5c28be83ed5e230095c202e39d88c8939e8992) refactor(application): keep a complete live post for reconciliation - [`f4a0781`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/f4a0781e175d71a7a6def4749cfd1aaf5a2c7d68) feat(application): preserve media order and reserve file paths - [`2057626`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/205762674149381cf4b4eaae99d1c15f6a2211cb) feat(application): merge post content with retained media history - [`1bed5b2`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/1bed5b2dfdb2d557b809f2188c60f9960e77671d) feat(application): track saved media files from disk snapshots - [`97d17c4`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/97d17c4b8d8809a3f0985e1ad6be7310f977ead9) refactor(application): share media names with file discovery - [`80e4b0b`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/80e4b0b9bc7c5b1d08994ddfb6d89ac9c1728056) feat(application): recover media state from existing files - [`027c83d`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/027c83d187d11ecb0182991929bebade80c8ef83) feat(application): reconcile posts and plan missing media downloads ### 📊 Changes **35 files changed** (+4004 additions, -108 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -0) ➕ `boosty_downloader/application/_media_order.py` (+30 -0) 📝 `boosty_downloader/application/filtering.py` (+11 -0) 📝 `boosty_downloader/application/mappers/link_header_text.py` (+4 -1) 📝 `boosty_downloader/application/mappers/list.py` (+12 -6) ➕ `boosty_downloader/application/mappers/live_post.py` (+292 -0) ➕ `boosty_downloader/application/media_paths.py` (+258 -0) ➕ `boosty_downloader/application/reconcile.py` (+289 -0) 📝 `boosty_downloader/cli/views/blog_overview.py` (+31 -3) 📝 `boosty_downloader/domain/stored_post.py` (+11 -3) 📝 `boosty_downloader/infrastructure/boosty_api/models/post/post_data_types/post_data_text.py` (+3 -1) 📝 `boosty_downloader/infrastructure/file_downloader.py` (+6 -7) 📝 `boosty_downloader/infrastructure/html_generator/renderer.py` (+4 -1) 📝 `boosty_downloader/infrastructure/html_generator/templates/list.html` (+0 -1) ➕ `boosty_downloader/infrastructure/media_filenames.py` (+20 -0) 📝 `boosty_downloader/infrastructure/path_sanitizer.py` (+25 -1) 📝 `boosty_downloader/infrastructure/post_caching/post_cache.py` (+4 -1) 📝 `boosty_downloader/infrastructure/post_media_downloader.py` (+2 -19) 📝 `scripts/api_schema/__main__.py` (+22 -17) 📝 `scripts/api_schema/schema_changes.py` (+17 -6) _...and 15 more files_ </details> ### 📄 Description ## 📝 Summary Add the core needed to save a complete post across partial downloads: text and images today, then video later. It combines the saved record, the current post and a disk snapshot to update media state and list missing downloads. ## 🔧 What changes - Map the full post body and media metadata, including uploads that are not ready to download. - Keep saved paths, timestamps and media removed by the author when refreshing a post. - Observe saved files and recover completed copies before planning downloads, with each path reserved for its owner. - Select ready missing media by content filter, restore option and unavailable IDs already checked during the run. - Share filename rules with the existing downloader, including portable handling of colons, backslashes and unusually long extensions. - Use named records, enums and small formatting helpers in the related CLI, HTML, cache and schema-report code. The new mapper and reconciliation API currently have test callers only. The existing downloader already uses the shared filename helpers; dry-run and download execution will use reconciliation in later changes. ## 🧪 Tests - `task ci` passed: 693 unit and local HTTP e2e tests, Ruff, formatting, Pyright, import boundaries and package builds. - Cases cover partial runs, edited posts, removed media, restored files, name collisions, lost access, unfinished uploads and retries. - Local HTTP tests verify that discovery finds files written by the actual downloader under the shared naming rules. - Independent code and task reviews passed. The branch merges cleanly with current `main`. - Live Boosty integration was not run: this worktree has no `.env`, and the new reconciliation API has no CLI entry point yet. ## ⏭️ Next Use reconciliation in `download_plan.py` to count exact missing media during a dry run. Later changes connect record persistence, download execution and full-page rendering. ## ✅ Checklist - [ ] Tested live on a real blog (feature or fix) - [x] Updated CLI `--help` and README if needed (no CLI interface changes) - [x] Updated `docs/` if the download flow or dev process changed (neither changes in this PR) - [x] Added a line to `## Unreleased` in CHANGELOG.md (or the `ci/skip-changelog` label if not user-facing) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:49:00 +00:00