[PR #51] [MERGED] More intelligent caching layer #79

New issue

Closed

opened 2026-10-13 19:46:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 19:46:56 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Glitchy-Sheep/boosty-downloader/pull/51
Author: @Glitchy-Sheep
Created: 8/1/2025
Status: ✅ Merged
Merged: 8/1/2025
Merged by: @Glitchy-Sheep

Base: dev ← Head: feat/better-cache-layer

📝 Commits (10+)

f39cc73 feat: 📝 Update Makefile: add analysis target + utf-8 support to use pipe-like cmds
b9002e3 feat: ✨ Intelligent caching, refactor
f8fd30e refactor: 🎨 Format the code
e348e74 refactor: ♻️ Remove old comments
6a2f84a Fix typo
6ae33d1 Better doc comment
922d562 refactor: ✏️ Fix var name typo
703edc9 Merge branch 'feat/better-cache-layer' of github.com:Glitchy-Sheep/boosty-downloader into feat/better-cache-layer
6069d45 Add doc string to the post cache save method
eb1eb92 refactor: ♻️ More specific imports

📊 Changes

11 files changed (+465 additions, -149 deletions)

View changed files

📝 Makefile (+21 -7)
📝 boosty_downloader/main.py (+29 -10)
➖ boosty_downloader/src/caching/post_cache.py (+0 -104)
📝 boosty_downloader/src/download_manager/download_manager.py (+13 -19)
📝 boosty_downloader/src/download_manager/download_manager_config.py (+2 -2)
📝 boosty_downloader/src/download_manager/storage/__init__.py (+0 -0)
➕ boosty_downloader/src/download_manager/storage/post_cache.py (+230 -0)
📝 boosty_downloader/src/loggers/base.py (+1 -4)
📝 boosty_downloader/src/loggers/logger_instances.py (+2 -2)
📝 poetry.lock (+166 -1)
📝 pyproject.toml (+1 -0)

📄 Description

📌 More intelligent caching layer

📝 Description

Before this change:

The utility cached posts as soon as any user-requested content was downloaded.

This led to undesirable behavior — if the user later tried to download content using filters that hadn't been applied before, the post would still be skipped as cached.

After this change:

The cache now tracks which types of content were downloaded and which were not. It also checks the updated_at fields to ensure that even previously downloaded content is still up-to-date.

🔄 Changelog

✨ Added:
- Better caching behavior
- TZ Aware datetime saving (using isoformat)
- DB-related exception handling to warn user
- Makefile target for running analysis faster
🔄 Changed:
- Now cache is the downloader storage layer
- Better naming for the project logger
- Now db-layer code uses SQLAlchemy (more extensible solution)

✅ Checklist

Locally tested (make test and your own judgment)
Documentation updated (if necessary)
Code follows the project's style guidelines (make lint && make format)

⚠ Notes

Because this change adds db fields - this update will increase major version.
For this reason I also added db-exceptions check to make sure user is aware of what's going on.

I would like to add some migrations, but since they're really dynamic (cache can appear anywhere in the system), it's harder than usual.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Glitchy-Sheep/boosty-downloader/pull/51 **Author:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Created:** 8/1/2025 **Status:** ✅ Merged **Merged:** 8/1/2025 **Merged by:** [@Glitchy-Sheep](https://github.com/Glitchy-Sheep) **Base:** `dev` ← **Head:** `feat/better-cache-layer` --- ### 📝 Commits (10+) - [`f39cc73`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/f39cc7343bdf0cc59e2f55e8ef9a814df80ac4c6) feat: :memo: Update Makefile: add analysis target + utf-8 support to use pipe-like cmds - [`b9002e3`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/b9002e301e454017653fdc61703e55d124db2c27) feat: :sparkles: Intelligent caching, refactor - [`f8fd30e`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/f8fd30eec34e69a32ca43d67f5bae9239dcb9dbc) refactor: :art: Format the code - [`e348e74`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/e348e74ffac69ebf095849a8f98155fd9707a8de) refactor: :recycle: Remove old comments - [`6a2f84a`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/6a2f84ae2233ce1eb8f3f3741070b60d875e8d50) Fix typo - [`6ae33d1`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/6ae33d1f3f6088ac4c55d60e2e59dc9c15d8441d) Better doc comment - [`922d562`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/922d56282cfc942ee7d451ee2798c0a2ff569833) refactor: :pencil2: Fix var name typo - [`703edc9`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/703edc94e2138f2c06fd6904488e9eff5c96a9f7) Merge branch 'feat/better-cache-layer' of github.com:Glitchy-Sheep/boosty-downloader into feat/better-cache-layer - [`6069d45`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/6069d45b21a2fa743f0c2593cedbd8f57eea8a4e) Add doc string to the post cache save method - [`eb1eb92`](https://github.com/Glitchy-Sheep/boosty-downloader/commit/eb1eb92a408e8714b82b29fb73c286a6d2e11883) refactor: :recycle: More specific imports ### 📊 Changes **11 files changed** (+465 additions, -149 deletions) <details> <summary>View changed files</summary> 📝 `Makefile` (+21 -7) 📝 `boosty_downloader/main.py` (+29 -10) ➖ `boosty_downloader/src/caching/post_cache.py` (+0 -104) 📝 `boosty_downloader/src/download_manager/download_manager.py` (+13 -19) 📝 `boosty_downloader/src/download_manager/download_manager_config.py` (+2 -2) 📝 `boosty_downloader/src/download_manager/storage/__init__.py` (+0 -0) ➕ `boosty_downloader/src/download_manager/storage/post_cache.py` (+230 -0) 📝 `boosty_downloader/src/loggers/base.py` (+1 -4) 📝 `boosty_downloader/src/loggers/logger_instances.py` (+2 -2) 📝 `poetry.lock` (+166 -1) 📝 `pyproject.toml` (+1 -0) </details> ### 📄 Description # 📌 More intelligent caching layer ## 📝 Description **Before this change:** - The utility cached posts as soon as any user-requested content was downloaded. **This led to undesirable behavior** — if the user later tried to download content using filters that hadn't been applied before, the post would still be skipped as cached. **After this change:** - The cache now tracks which types of content were downloaded and which were not. It also checks the `updated_at` fields to ensure that even previously downloaded content is still up-to-date. ## 🔄 Changelog  - **✨ Added:** - Better caching behavior - TZ Aware datetime saving (using isoformat) - DB-related exception handling to warn user - Makefile target for running analysis faster - **🔄 Changed:** - Now cache is the downloader storage layer - Better naming for the project logger - Now db-layer code uses SQLAlchemy (more extensible solution) ## ✅ Checklist - [x] Locally tested (`make test` and your own judgment) - [ ] Documentation updated (if necessary) - [x] Code follows the project's style guidelines (`make lint && make format`) ## ⚠ Notes Because this change adds db fields - this update will increase major version. For this reason I also added db-exceptions check to make sure user is aware of what's going on. I would like to add some migrations, but since they're really dynamic (cache can appear anywhere in the system), it's harder than usual. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-13 19:46:56 +00:00