[PR #140] [MERGED] [downloader] Fit every generated name into the filesystem limit #144

New issue

Closed

opened 2026-10-13 19:47:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:47:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/140
Author: @Glitchy-Sheep
Created: 8/14/2026
Status: ✅ Merged
Merged: 8/14/2026
Merged by: @Glitchy-Sheep

Base: main ← Head: fix/long-filenames

📝 Commits (6)

2c05aad fix: Add byte-aware filename sanitizer and a path-limits reference
1e79644 fix: Compose post folder names through the byte-aware sanitizer
12420c5 fix: Keep file extensions and video ids when truncating long filenames
3fdecbb fix: Cap external video titles so yt-dlp names fit the filesystem limit
8a5609d feat: Hint at a shorter destination path on OS path-length errors
3013f2a docs: Note the long-filename fix in the changelog

📊 Changes

14 files changed (+489 additions, -39 deletions)

View changed files

📝 CHANGELOG.md (+8 -0)
📝 boosty_downloader/src/application/use_cases/download_all_posts.py (+13 -15)
📝 boosty_downloader/src/application/use_cases/download_single_post.py (+29 -1)
📝 boosty_downloader/src/application/use_cases/download_specific_post.py (+8 -8)
📝 boosty_downloader/src/infrastructure/external_videos_downloader/external_videos_downloader.py (+14 -2)
📝 boosty_downloader/src/infrastructure/file_downloader.py (+20 -7)
📝 boosty_downloader/src/infrastructure/path_sanitizer.py (+72 -6)
➕ docs/concepts/filesystem-path-limits.md (+70 -0)
➕ test/unit/external_videos_downloader/title_limit_test.py (+29 -0)
➕ test/unit/file_downloader/filename_truncation_test.py (+42 -0)
➕ test/unit/path_sanitizer/path_sanitizer_test.py (+82 -0)
➕ test/unit/path_sanitizer/path_too_long_error_test.py (+42 -0)
📝 test/unit/use_cases/download_all_posts_test.py (+18 -0)
➕ test/unit/use_cases/post_directory_name_test.py (+42 -0)

📄 Description

Closes #93

Summary

Long post titles crashed the download with OSError: File name too long. Filesystems cap one name at 255 bytes, and Cyrillic takes 2 bytes per letter in UTF-8 - so a title of 128 Russian letters was already over the limit. Now every name the app generates is cut to a 240-byte budget, and the parts that matter - the post date, the dedup id and the file extension - always survive the cut.

Problem

Post folders, file names and yt-dlp video names were built from titles with no length limit at all
The old sanitizer only removed <>:"/\|?*; newlines from real post titles stayed in folder names (illegal on Windows), and a title of pure emoji produced a hidden file like .mp4
The community fix (PR #86) truncated the composed string from the end - that would eat the (id8) dedup tail and file extensions, reviving #104 and #75
Two download paths named the same post differently: download added (id8), --post-url did not
The .replace('.', '') hack mangled titles: "v2.0" became "v20"

Fix

One policy function sanitize_filename(name, suffix, max_bytes): NFC normalization, control chars and unsafe chars removed, trailing dots and spaces stripped, Windows reserved names prefixed, byte truncation that never splits a UTF-8 char - and a suffix (extension or dedup id) that never gets truncated away
Both download paths compose the folder as date - title (id8) through one helper; only the title part is ever cut
Author file names are cut by the stem, their own extension survives; app-built names reserve room for the guessed extension
External video titles leave room for the extension yt-dlp appends
When the full path is still too long for the OS (deep destination folder), the error message now advises moving the destination closer to the drive root - the detector recognizes ENAMETOOLONG and Windows WinError 206 even under error wrappers
docs/concepts/filesystem-path-limits.md: a practical reference of per-system limits, with real paths that fit and fail

Before / After

Before: a post titled with 130 Cyrillic letters fails on Linux with a raw OSError: [Errno 36] File name too long, lands in the failed log with no advice, and the same post gets different folders via download and --post-url.
After: the folder is 2026-08-14 - яяя...я (a2dd6942) at 240 bytes, both paths agree on it, archive.zip cut from a long title still ends with .zip, and a genuine path-length error tells the user what to do.

Compatibility

Existing downloads are untouched: the cache stores only post uuids and flags, never paths; folders are never renamed or deleted
A new-style folder name appears only when an updated post is re-downloaded; unchanged cached posts are skipped exactly as before

Verification

Live on APFS: the #93 repro ('я' * 300 as a title) fails mkdir with a raw name and succeeds with the composed one at exactly 240 bytes with the id tail intact
Live probing also pinned the real APFS limit: 255 UTF-16 units, not bytes or characters - documented in the new reference
The path-length detector fires on a real FS error (errno 63) and stays silent on unrelated errors
29 new unit tests across the sanitizer, folder composition, file truncation, video titles and the error hint; the #104 and #75 regression tests pass unchanged; 100 tests total
task check (ruff, format, pyright) and task build green

Notes

eCryptfs (143-byte names) is knowingly out of support
The full-path limit (Windows 260 by default) cannot be fixed by a name budget - the user picks the base folder; the error hint plus the docs reference cover it
Found during verification: on POSIX a destination deeper than ~490 bytes breaks the SQLite cache first (its own 512-byte path limit) with a misleading "cache outdated" message - exotic, noted for a possible follow-up

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/140 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 8/14/2026 **Status:** ✅ Merged **Merged:** 8/14/2026 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `main` ← **Head:** `fix/long-filenames` --- ### 📝 Commits (6) - [`2c05aad`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/2c05aadf86f9050aaa420ab0b9a729d745e9c6e4) fix: Add byte-aware filename sanitizer and a path-limits reference - [`1e79644`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/1e79644e754ad2c0adf0a6492c1377587b4c9ff5) fix: Compose post folder names through the byte-aware sanitizer - [`12420c5`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/12420c57679c783412579f2202c007e080a3d806) fix: Keep file extensions and video ids when truncating long filenames - [`3fdecbb`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/3fdecbb87c4261bd1df75a8e664faad4ea7b761f) fix: Cap external video titles so yt-dlp names fit the filesystem limit - [`8a5609d`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/8a5609d8231f174118229b00ddac617492090f35) feat: Hint at a shorter destination path on OS path-length errors - [`3013f2a`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/3013f2afbf3b4bd8a6b2ee8f448b7125b5e87d37) docs: Note the long-filename fix in the changelog ### 📊 Changes **14 files changed** (+489 additions, -39 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+8 -0) 📝 `boosty_downloader/src/application/use_cases/download_all_posts.py` (+13 -15) 📝 `boosty_downloader/src/application/use_cases/download_single_post.py` (+29 -1) 📝 `boosty_downloader/src/application/use_cases/download_specific_post.py` (+8 -8) 📝 `boosty_downloader/src/infrastructure/external_videos_downloader/external_videos_downloader.py` (+14 -2) 📝 `boosty_downloader/src/infrastructure/file_downloader.py` (+20 -7) 📝 `boosty_downloader/src/infrastructure/path_sanitizer.py` (+72 -6) ➕ `docs/concepts/filesystem-path-limits.md` (+70 -0) ➕ `test/unit/external_videos_downloader/title_limit_test.py` (+29 -0) ➕ `test/unit/file_downloader/filename_truncation_test.py` (+42 -0) ➕ `test/unit/path_sanitizer/path_sanitizer_test.py` (+82 -0) ➕ `test/unit/path_sanitizer/path_too_long_error_test.py` (+42 -0) 📝 `test/unit/use_cases/download_all_posts_test.py` (+18 -0) ➕ `test/unit/use_cases/post_directory_name_test.py` (+42 -0) </details> ### 📄 Description Closes #93 ## Summary Long post titles crashed the download with `OSError: File name too long`. Filesystems cap one name at 255 bytes, and Cyrillic takes 2 bytes per letter in UTF-8 - so a title of 128 Russian letters was already over the limit. Now every name the app generates is cut to a 240-byte budget, and the parts that matter - the post date, the dedup id and the file extension - always survive the cut. ## Problem - Post folders, file names and yt-dlp video names were built from titles with no length limit at all - The old sanitizer only removed `<>:"/\|?*`; newlines from real post titles stayed in folder names (illegal on Windows), and a title of pure emoji produced a hidden file like `.mp4` - The community fix (PR #86) truncated the composed string from the end - that would eat the `(id8)` dedup tail and file extensions, reviving #104 and #75 - Two download paths named the same post differently: `download` added `(id8)`, `--post-url` did not - The `.replace('.', '')` hack mangled titles: "v2.0" became "v20" ## Fix - One policy function `sanitize_filename(name, suffix, max_bytes)`: NFC normalization, control chars and unsafe chars removed, trailing dots and spaces stripped, Windows reserved names prefixed, byte truncation that never splits a UTF-8 char - and a `suffix` (extension or dedup id) that never gets truncated away - Both download paths compose the folder as `date - title (id8)` through one helper; only the title part is ever cut - Author file names are cut by the stem, their own extension survives; app-built names reserve room for the guessed extension - External video titles leave room for the extension yt-dlp appends - When the full path is still too long for the OS (deep destination folder), the error message now advises moving the destination closer to the drive root - the detector recognizes `ENAMETOOLONG` and Windows `WinError 206` even under error wrappers - `docs/concepts/filesystem-path-limits.md`: a practical reference of per-system limits, with real paths that fit and fail ## Before / After **Before:** a post titled with 130 Cyrillic letters fails on Linux with a raw `OSError: [Errno 36] File name too long`, lands in the failed log with no advice, and the same post gets different folders via `download` and `--post-url`. **After:** the folder is `2026-08-14 - яяя...я (a2dd6942)` at 240 bytes, both paths agree on it, `archive.zip` cut from a long title still ends with `.zip`, and a genuine path-length error tells the user what to do. ## Compatibility - Existing downloads are untouched: the cache stores only post uuids and flags, never paths; folders are never renamed or deleted - A new-style folder name appears only when an updated post is re-downloaded; unchanged cached posts are skipped exactly as before ## Verification - Live on APFS: the #93 repro (`'я' * 300` as a title) fails `mkdir` with a raw name and succeeds with the composed one at exactly 240 bytes with the id tail intact - Live probing also pinned the real APFS limit: 255 UTF-16 units, not bytes or characters - documented in the new reference - The path-length detector fires on a real FS error (`errno 63`) and stays silent on unrelated errors - 29 new unit tests across the sanitizer, folder composition, file truncation, video titles and the error hint; the #104 and #75 regression tests pass unchanged; 100 tests total - `task check` (ruff, format, pyright) and `task build` green ## Notes - eCryptfs (143-byte names) is knowingly out of support - The full-path limit (Windows 260 by default) cannot be fixed by a name budget - the user picks the base folder; the error hint plus the docs reference cover it - Found during verification: on POSIX a destination deeper than ~490 bytes breaks the SQLite cache first (its own 512-byte path limit) with a misleading "cache outdated" message - exotic, noted for a possible follow-up --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:47:55 +00:00