mirror of
https://github.com/deepseek-ai/DeepSeek-R1.git
synced 2026-11-05 00:47:42 +00:00
[GH-ISSUE #203] MoE only load activated expert(s) to GPU while rest non-used experts are not loaded (to CPU/GPU) for DeekSeek-R1 or V3 Inference on consumer GPU #143
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-R1#143
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @marvin-0042 on GitHub (Jan 30, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/203
Running DeekSeep-R1 or V3 inference needs 8xH100 80GB due to huge memory footprint, and it's very challenging to do R1 or V3 inference on single consumer GPU RAM (e.g. 24GB 4090) + limited CPU memory (say 32GB) with 685B MoE params even with low-bit quantization.
But since V3 and R1 has only 37B activated params (INT4 37B weights is 18.5GB), is it possible for the MoE inference to only load the 37B "activated experts (s)" related weights to GPU mem, and leave other non-activated or non-used expert's weight some in CPU memory(e.g.32GB), but majority weights on disk because CPU memory is also limited, and only load/unload these weights when in use ?
I'm wondering if similar features is available or WIP inside DeepSeek-V3 github or any popular inference frameworks ?
Really appreciate your help!
@dinithaw commented on GitHub (Feb 1, 2025):
Just install LM studio. It will show what can run on your hardware smoothly. It's not recommend to use such llms with gigantic number of parameters on consumer hardware
@RArchered commented on GitHub (Feb 12, 2025):
maybe not work. to predict next token, model such as deepseek r1 will choose several experts that should be loaded on GPU, then go on, predicting next token, maybe other experts will be used, we should load these experts from memory/disks to GPU memory that consumes much time(several seconds at least). therefore, outputing one token needs several seconds, too slow to work.
@github-actions[bot] commented on GitHub (Apr 5, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@github-actions[bot] commented on GitHub (May 8, 2025):
false