theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-03-15 00:07:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #393 from mefich/main

Browse Source 
Unloading vision model of VLMs for Exllamav3 backend
This commit is contained in:
Brian
2025-11-19 22:46:59 -05:00
committed by GitHub
parent fece4791ad 37aea9de83
commit df724fdc78
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
backends/exllamav3/model.py
View File

@@ -563,6 +563,10 @@ class ExllamaV3Container(BaseModelContainer):
self.draft_config = None self.draft_config = None
self.draft_cache = None self.draft_cache = None
if self.use_vision:
self.vision_model.unload()
self.vision_model = None
# Cleanup the generator from any pending jobs # Cleanup the generator from any pending jobs
if self.generator is not None: if self.generator is not None:
await self.generator.close() await self.generator.close()