[GH-ISSUE #34] Adding support for quantization and fast inference server #22

New issue

Open

opened 2026-10-26 21:08:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:08:54 +00:00

Owner

Originally created by @rostandk on GitHub (Mar 20, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL/issues/34

DeepSeek-VL 7B seems to be the awesome vision model that nobody is talking about. It is better than Llava 1.6 34B in VQA and OCR. It consistently return JSON when asked and just seems to have a great textual capability. Any recommendation on how to wrap DeepSeek-VL in a fast inference server and take for spin at scale ?

There are some issues opened here:
https://github.com/InternLM/lmdeploy/issues/1321
https://github.com/sgl-project/sglang/issues/297

Originally created by @rostandk on GitHub (Mar 20, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL/issues/34 DeepSeek-VL 7B seems to be the awesome vision model that nobody is talking about. It is better than Llava 1.6 34B in VQA and OCR. It consistently return JSON when asked and just seems to have a great textual capability. Any recommendation on how to wrap DeepSeek-VL in a fast inference server and take for spin at scale ? There are some issues opened here: https://github.com/InternLM/lmdeploy/issues/1321 https://github.com/sgl-project/sglang/issues/297

bsakaguchi referenced this issue

2026-10-26 21:09:20 +00:00

[PR #22] feat: add mps support #67