[PR #115] [MERGED] [boosty-api] Survive unknown Boosty content instead of crashing the download #120

New issue

Closed

opened 2026-10-13 19:47:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:47:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/115
Author: @Glitchy-Sheep
Created: 8/8/2026
Status: ✅ Merged
Merged: 8/8/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: fix/tolerant-api-validation

📝 Commits (6)

a6727fa fix: 🛡️ Tolerate unknown ok_video url types from Boosty
c841736 Keep unknown Boosty values as named types instead of crashing
18211d1 docs: 📝 Describe download flow and the Tolerant Reader concept
9ec0e21 feat: ✨ Human-readable validation error details
e8aee24 fix: 🛡️ Skip broken posts with a warning instead of failing the page
d48b11e feat: ✨ Final run summary of skipped posts and unknown content

📊 Changes

28 files changed (+1129 additions, -46 deletions)

View changed files

📝 CHANGELOG.md (+2 -0)
📝 boosty_downloader/main.py (+8 -3)
📝 boosty_downloader/src/application/mappers/list.py (+2 -1)
📝 boosty_downloader/src/application/mappers/post_mapper.py (+17 -0)
📝 boosty_downloader/src/application/use_cases/check_total_posts.py (+29 -0)
📝 boosty_downloader/src/application/use_cases/download_all_posts.py (+36 -0)
📝 boosty_downloader/src/application/use_cases/download_specific_post.py (+82 -26)
📝 boosty_downloader/src/infrastructure/boosty_api/core/client.py (+26 -4)
📝 boosty_downloader/src/infrastructure/boosty_api/models/post/base_post_data.py (+31 -2)
📝 boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/__init__.py (+2 -0)
📝 boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_list.py (+38 -3)
📝 boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_ok_video.py (+16 -2)
➕ boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_unknown.py (+14 -0)
📝 boosty_downloader/src/infrastructure/boosty_api/models/post/posts_request.py (+26 -5)
➕ boosty_downloader/src/infrastructure/boosty_api/models/unknown_content.py (+64 -0)
➕ boosty_downloader/src/infrastructure/boosty_api/models/unknown_value.py (+31 -0)
➕ boosty_downloader/src/infrastructure/boosty_api/utils/validation_errors.py (+91 -0)
➕ docs/concepts/boosty/download-flow.md (+32 -0)
➕ docs/concepts/boosty/tolerant-reader.md (+52 -0)
📝 test/unit/boosty_api/client_test.py (+61 -0)

...and 8 more files

📄 Description

Summary

Boosty changes its API without notice, and one unknown word used to kill the whole download. The client now follows the Tolerant Reader pattern: unknown data is kept, skipped where needed, and reported to the user with exact paths.

Problem

Boosty added new ondemand_dash / ondemand_hls video url types (Aug 2026), and every check/download run died with a validation error
One broken post failed the whole page of up to 100 posts
Validation details printed as a raw wall of pydantic dicts
Nothing told the user what exactly the client didn't understand, so there was nothing to report

Fix

Unknown data exists only as two named types: BoostyUnknownValue(raw=...) for values outside a known enum, BoostyPostDataUnknownDTO for whole chunks
Fields with a fixed set of values use one pattern: Enum | UnknownValue, the enum stays the point of truth, the raw word is kept
A known chunk with a broken body still fails validation, so real breakage is not masked as "unknown"
Posts validate one by one: a broken post becomes a warning with its title, id and problems, the rest of the page downloads
One generic walk (collect_unknown_content) finds every unknown in a parsed post as an API path, with no registry to keep in sync
check and download end with a summary of everything not understood, asking to report it at GitHub issues
download --post-url says the real reason when the target post itself can't be parsed - it used to end with a misleading "not found"
New docs: docs/concepts/boosty/ (download flow + Tolerant Reader)

Before / After

Before: check -u <author> on a blog with one new video url type crashed with a wall of pydantic error dicts.

After: the run finishes and ends with:

Some content was not understood by this version of the downloader:
 Unknown content (downloaded around, not lost):
  - data[8].playerUrls[2].type = 'ondemand_dash'
  - data[8].playerUrls[8].type = 'ondemand_hls'
Please report this at https://github.com/Glitchy-Sheep/boosty-downloader/issues so the client can be updated.

Tests

test_unknown_url_type_keeps_raw_word, test_known_url_type_still_parses_exactly - tolerant enum parsing, and the catch-all never swallows known values
test_unknown_chunk_type_parses_as_unknown, test_known_chunk_with_broken_body_still_fails - chunk fallback triggers on unknown tags only
test_wrapper_is_found_at_any_depth, test_clean_tree_yields_nothing - the generic walk
test_broken_post_is_skipped_and_page_survives, test_broken_pagination_still_fails_the_page - per-post validation boundaries
test_enum_error_shows_the_offending_word, test_nested_loc_renders_as_api_path - readable validation errors
test_skipped_post_warns_inline_and_run_summary_reports_everything, test_run_summary_is_none_when_clean - the final summary
Checked live against the real API: check and download on a blog with the new video url types finish and print the summary above

Notes

The searched-post-is-broken path in download --post-url and clean-run silence are covered by tests; the incident blog itself was the live testbed

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/115 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 8/8/2026 **Status:** ✅ Merged **Merged:** 8/8/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `fix/tolerant-api-validation` --- ### 📝 Commits (6) - [`a6727fa`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/a6727fa0342979f59d10c61e9b22dbf74e7fa89d) fix: :shield: Tolerate unknown ok_video url types from Boosty - [`c841736`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/c8417362e7988c44b05a519ad447af3238c28a45) Keep unknown Boosty values as named types instead of crashing - [`18211d1`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/18211d10e78b43b5d65de2873ec10b143598eaba) docs: :memo: Describe download flow and the Tolerant Reader concept - [`9ec0e21`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/9ec0e2150f416857634267fed91507ab13e3b617) feat: :sparkles: Human-readable validation error details - [`e8aee24`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/e8aee24905aa3d89153f07c75fe83b603e633f6b) fix: :shield: Skip broken posts with a warning instead of failing the page - [`d48b11e`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/d48b11e9f13c6fc364f8662fa008612c434a0fc0) feat: :sparkles: Final run summary of skipped posts and unknown content ### 📊 Changes **28 files changed** (+1129 additions, -46 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+2 -0) 📝 `boosty_downloader/main.py` (+8 -3) 📝 `boosty_downloader/src/application/mappers/list.py` (+2 -1) 📝 `boosty_downloader/src/application/mappers/post_mapper.py` (+17 -0) 📝 `boosty_downloader/src/application/use_cases/check_total_posts.py` (+29 -0) 📝 `boosty_downloader/src/application/use_cases/download_all_posts.py` (+36 -0) 📝 `boosty_downloader/src/application/use_cases/download_specific_post.py` (+82 -26) 📝 `boosty_downloader/src/infrastructure/boosty_api/core/client.py` (+26 -4) 📝 `boosty_downloader/src/infrastructure/boosty_api/models/post/base_post_data.py` (+31 -2) 📝 `boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/__init__.py` (+2 -0) 📝 `boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_list.py` (+38 -3) 📝 `boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_ok_video.py` (+16 -2) ➕ `boosty_downloader/src/infrastructure/boosty_api/models/post/post_data_types/post_data_unknown.py` (+14 -0) 📝 `boosty_downloader/src/infrastructure/boosty_api/models/post/posts_request.py` (+26 -5) ➕ `boosty_downloader/src/infrastructure/boosty_api/models/unknown_content.py` (+64 -0) ➕ `boosty_downloader/src/infrastructure/boosty_api/models/unknown_value.py` (+31 -0) ➕ `boosty_downloader/src/infrastructure/boosty_api/utils/validation_errors.py` (+91 -0) ➕ `docs/concepts/boosty/download-flow.md` (+32 -0) ➕ `docs/concepts/boosty/tolerant-reader.md` (+52 -0) 📝 `test/unit/boosty_api/client_test.py` (+61 -0) _...and 8 more files_ </details> ### 📄 Description ## Summary Boosty changes its API without notice, and one unknown word used to kill the whole download. The client now follows the Tolerant Reader pattern: unknown data is kept, skipped where needed, and reported to the user with exact paths. ## Problem - Boosty added new `ondemand_dash` / `ondemand_hls` video url types (Aug 2026), and every `check`/`download` run died with a validation error - One broken post failed the whole page of up to 100 posts - Validation details printed as a raw wall of pydantic dicts - Nothing told the user what exactly the client didn't understand, so there was nothing to report ## Fix - Unknown data exists only as two named types: `BoostyUnknownValue(raw=...)` for values outside a known enum, `BoostyPostDataUnknownDTO` for whole chunks - Fields with a fixed set of values use one pattern: `Enum | UnknownValue`, the enum stays the point of truth, the raw word is kept - A known chunk with a broken body still fails validation, so real breakage is not masked as "unknown" - Posts validate one by one: a broken post becomes a warning with its title, id and problems, the rest of the page downloads - One generic walk (`collect_unknown_content`) finds every unknown in a parsed post as an API path, with no registry to keep in sync - `check` and `download` end with a summary of everything not understood, asking to report it at GitHub issues - `download --post-url` says the real reason when the target post itself can't be parsed - it used to end with a misleading "not found" - New docs: `docs/concepts/boosty/` (download flow + Tolerant Reader) ## Before / After **Before:** `check -u <author>` on a blog with one new video url type crashed with a wall of pydantic error dicts. **After:** the run finishes and ends with: ``` Some content was not understood by this version of the downloader: Unknown content (downloaded around, not lost): - data[8].playerUrls[2].type = 'ondemand_dash' - data[8].playerUrls[8].type = 'ondemand_hls' Please report this at https://github.com/Glitchy-Sheep/boosty-downloader/issues so the client can be updated. ``` ## Tests - `test_unknown_url_type_keeps_raw_word`, `test_known_url_type_still_parses_exactly` - tolerant enum parsing, and the catch-all never swallows known values - `test_unknown_chunk_type_parses_as_unknown`, `test_known_chunk_with_broken_body_still_fails` - chunk fallback triggers on unknown tags only - `test_wrapper_is_found_at_any_depth`, `test_clean_tree_yields_nothing` - the generic walk - `test_broken_post_is_skipped_and_page_survives`, `test_broken_pagination_still_fails_the_page` - per-post validation boundaries - `test_enum_error_shows_the_offending_word`, `test_nested_loc_renders_as_api_path` - readable validation errors - `test_skipped_post_warns_inline_and_run_summary_reports_everything`, `test_run_summary_is_none_when_clean` - the final summary - Checked live against the real API: `check` and `download` on a blog with the new video url types finish and print the summary above ## Notes - The searched-post-is-broken path in `download --post-url` and clean-run silence are covered by tests; the incident blog itself was the live testbed --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:47:35 +00:00