Glitchy-Sheep/boosty-downloader

Fork 0

mirror of https://github.com/Glitchy-Sheep/boosty-downloader.git synced 2026-10-14 21:33:23 +00:00

[GH-ISSUE #95] Feature Request: --cache-dir flag for custom cache database location #21

New issue

Closed

opened 2026-10-13 19:45:52 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-13 19:45:52 +00:00

Owner

Originally created by @Mifody-07 on GitHub (Mar 9, 2026).
Original GitHub issue: https://github.com/Glitchy-Sheep/boosty-downloader/issues/95

Originally assigned to: @Glitchy-Sheep on GitHub.

Feature Request: `--cache-dir` flag to specify custom cache database location

The Story (or: how a simple task turned into patching source code)

I've been using boosty-downloader on my laptop — it worked great. But at some point I decided to move it into a Proxmox LXC container so downloads would run independently in the background while I do other things, without keeping a terminal open on my laptop.

The setup looked reasonable:

LXC container running Ubuntu
Synology NAS mounted via CIFS/SMB inside the container
boosty-downloader saving files directly to the NAS share
post_cache.db ending up on the NAS share too (since it's created next to the output directory)

And then this happened:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked

Why this happens

SQLite is not a server — it's just a file. To prevent data corruption when writing, it uses OS-level file locking: it asks the operating system to lock the file while writing, so nothing else can touch it.

This works perfectly on local filesystems (ext4, btrfs, ntfs).

But CIFS/SMB is a network protocol. When Linux sends a file lock request over CIFS — it goes over the network to the Synology NAS. Synology either ignores it or acknowledges it without actually enforcing it. SQLite believes the lock was acquired, tries to write, and gets a conflict.

Result: database is locked — even when there's only one process running.

Why not just fix the network storage?

The obvious alternatives:

Run boosty-downloader directly on the NAS — Synology DSM is a closed platform. No pip, limited Python, and Synology explicitly warns that data outside shared folders may be deleted on system updates. Not viable.
Use NFS instead of CIFS — NFS supports POSIX file locking properly, SQLite would work. But I already have CIFS shares set up for everything (Windows clients, media players, etc). Reconfiguring the entire storage stack just to fix one SQLite file is overengineering.
Use iSCSI — NAS exposes a block device, container formats it as ext4. SQLite works perfectly. But again — massive overengineering for a single cache file.
Mount cache on tmpfs (RAM) — SQLite works, but cache is lost on every container restart, forcing full re-download every time.

All roads lead to the same conclusion: the cache database needs to be on a local disk, separate from the output directory.

Current workaround

Manually patching app_environment.py:

# Original
post_cache = SQLitePostCache(
    destination=self.target_directory / self.author_name,
    logger=self.logger,
)

# Patched
_local_cache_dir = Path('/root/boosty-cache') / self.author_name
_local_cache_dir.mkdir(parents=True, exist_ok=True)
post_cache = SQLitePostCache(
    destination=_local_cache_dir,
    logger=self.logger,
)

This works — but it's a hack that breaks on every pip install --upgrade boosty-downloader.

Proposed Solution

Add a --cache-dir CLI flag and corresponding config.yaml option:

# CLI
boosty-downloader -u author -o /mnt/nas/boosty --cache-dir /root/boosty-cache

# config.yaml
downloading_settings:
  target_directory: '/mnt/nas/boosty'
  cache_directory: '/root/boosty-cache'  # optional, defaults to target_directory

If --cache-dir is not specified — behavior stays exactly as now (cache next to output). No breaking changes.

Who needs this

Anyone running boosty-downloader with output to network storage:

Synology / QNAP / TrueNAS NAS owners
Proxmox / Docker setups with CIFS-mounted shares
Any homelab setup where "download to NAS" is the natural use case

This is not an edge case — it's probably the most common production setup for a tool like this.

I am ready to try implementing this and submit a pull request if you're interested.

Originally created by @Mifody-07 on GitHub (Mar 9, 2026). Original GitHub issue: https://github.com/Glitchy-Sheep/boosty-downloader/issues/95 Originally assigned to: @Glitchy-Sheep on GitHub. ## Feature Request: `--cache-dir` flag to specify custom cache database location ### The Story (or: how a simple task turned into patching source code) I've been using boosty-downloader on my laptop — it worked great. But at some point I decided to move it into a **Proxmox LXC container** so downloads would run independently in the background while I do other things, without keeping a terminal open on my laptop. The setup looked reasonable: - LXC container running Ubuntu - Synology NAS mounted via **CIFS/SMB** inside the container - boosty-downloader saving files directly to the NAS share - `post_cache.db` ending up on the NAS share too (since it's created next to the output directory) And then this happened: ``` sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked ``` --- ### Why this happens SQLite is not a server — it's just a file. To prevent data corruption when writing, it uses **OS-level file locking**: it asks the operating system to lock the file while writing, so nothing else can touch it. This works perfectly on local filesystems (ext4, btrfs, ntfs). **But CIFS/SMB is a network protocol.** When Linux sends a file lock request over CIFS — it goes over the network to the Synology NAS. Synology either ignores it or acknowledges it without actually enforcing it. SQLite believes the lock was acquired, tries to write, and gets a conflict. Result: `database is locked` — even when there's only one process running. --- ### Why not just fix the network storage? The obvious alternatives: - **Run boosty-downloader directly on the NAS** — Synology DSM is a closed platform. No pip, limited Python, and Synology explicitly warns that data outside shared folders may be deleted on system updates. Not viable. - **Use NFS instead of CIFS** — NFS supports POSIX file locking properly, SQLite would work. But I already have CIFS shares set up for everything (Windows clients, media players, etc). Reconfiguring the entire storage stack just to fix one SQLite file is **overengineering**. - **Use iSCSI** — NAS exposes a block device, container formats it as ext4. SQLite works perfectly. But again — massive overengineering for a single cache file. - **Mount cache on tmpfs (RAM)** — SQLite works, but cache is lost on every container restart, forcing full re-download every time. **All roads lead to the same conclusion: the cache database needs to be on a local disk, separate from the output directory.** --- ### Current workaround Manually patching `app_environment.py`: ```python # Original post_cache = SQLitePostCache( destination=self.target_directory / self.author_name, logger=self.logger, ) # Patched _local_cache_dir = Path('/root/boosty-cache') / self.author_name _local_cache_dir.mkdir(parents=True, exist_ok=True) post_cache = SQLitePostCache( destination=_local_cache_dir, logger=self.logger, ) ``` This works — but it's a hack that breaks on every `pip install --upgrade boosty-downloader`. --- ### Proposed Solution Add a `--cache-dir` CLI flag and corresponding `config.yaml` option: ```bash # CLI boosty-downloader -u author -o /mnt/nas/boosty --cache-dir /root/boosty-cache # config.yaml downloading_settings: target_directory: '/mnt/nas/boosty' cache_directory: '/root/boosty-cache' # optional, defaults to target_directory ``` If `--cache-dir` is not specified — behavior stays exactly as now (cache next to output). No breaking changes. --- ### Who needs this Anyone running boosty-downloader with output to network storage: - Synology / QNAP / TrueNAS NAS owners - Proxmox / Docker setups with CIFS-mounted shares - Any homelab setup where "download to NAS" is the natural use case This is not an edge case — it's probably the most common production setup for a tool like this. --- **I am ready to try implementing this and submit a pull request if you're interested.**

bsakaguchi

2026-10-13 19:45:52 +00:00

closed this issue
added the
enhancement
label

bsakaguchi commented

2026-10-13 19:45:54 +00:00

Author

Owner

@Glitchy-Sheep commented on GitHub (Apr 11, 2026):

Yeah, that's reasonable, I'll make this soon I think

@Glitchy-Sheep commented on GitHub (Apr 11, 2026): Yeah, that's reasonable, I'll make this soon I think

bsakaguchi commented

2026-10-13 19:45:54 +00:00

Author

Owner

@Glitchy-Sheep commented on GitHub (Apr 12, 2026):

Version 2.2.0 has been released with the feature you described - hope it helps! 🤟

@Glitchy-Sheep commented on GitHub (Apr 12, 2026): Version 2.2.0 has been released with the feature you described - hope it helps! 🤟

bsakaguchi referenced this issue

2026-10-13 19:46:34 +00:00

[PR #21] [CLOSED] chore(deps-dev): Bump pyright from 1.1.400 to 1.1.401 #51