mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1056] [Bug] v0.2.4post1 runs, but does not open port #744
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#744
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @createthis on GitHub (Apr 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1056
Checklist
Describe the bug
I run:
result:
Then nothing happens.
Reproduction
Environment
OS: ubuntu 24.04.02
cuda:
I have a 24gb 3090 gpu.
I have 768gb ram.
I'm running dual 9355 epyc CPUs, but I didn't specify
USE_NUMA=1during build because I have less than 1TB ram.@magikRUKKOLA commented on GitHub (Apr 5, 2025):
debian?
@createthis commented on GitHub (Apr 5, 2025):
ubuntu 24.04.02
@magikRUKKOLA commented on GitHub (Apr 5, 2025):
I bet your problem is somehow connected to the flashinfer that you installed afterwards. As far as I understood they modified the flashinfer and named it custom_flashinfer. So you have to use their implementation.
@createthis commented on GitHub (Apr 5, 2025):
When I run with:
It starts, but when I run a query I get this:
@magikRUKKOLA commented on GitHub (Apr 5, 2025):
have you used install.sh or the old build rules (from the prev release)?
@createthis commented on GitHub (Apr 5, 2025):
I used this guide: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/balance-serve.md
I also had to do this to get it to run: https://github.com/kvcache-ai/ktransformers/issues/1019#issuecomment-2781016513
@createthis commented on GitHub (Apr 5, 2025):
Ok, I got this working:
I had to do this first:
Then I re-ran the above and it worked.
@createthis commented on GitHub (Apr 5, 2025):
--backend_type balance_serveis still not working though.@magikRUKKOLA commented on GitHub (Apr 5, 2025):
You've done that, right?
(install.sh)
@createthis commented on GitHub (Apr 5, 2025):
Yes. I tried it manually just now and it had no affect.
@magikRUKKOLA commented on GitHub (Apr 5, 2025):
Would you please post the output of
?
@createthis commented on GitHub (Apr 5, 2025):
@magikRUKKOLA commented on GitHub (Apr 5, 2025):
what about
?
@createthis commented on GitHub (Apr 5, 2025):
@datouxia commented on GitHub (Apr 6, 2025):
Try to download DeepSeek/deepseek-v3-0324 directory small size files into a directory, which must use "DeepSeek-V3" as the end part of your model_path like this one:
"--model_path /data/DeepSeek-V3",
Otherwise the server process look likes blocked forever.
@createthis commented on GitHub (Apr 6, 2025):
@datouxia I downloaded the small files and tried this:
It just hangs. Here's
/data/DeepSeek-V3-0324/q4_files/Q4_K_M:@createthis commented on GitHub (Apr 6, 2025):
lol. I was debugging this with DeepSeek-R1 today and it rightly pointed out that I'm using Cuda 12.8 with Pytorch built for Cuda 12.6. 🤦
I'll fix it and report back.
UPDATE1:
I installed the nightly cu128 torch and rebuilt everything:
But it still hung.
Will try downgrading to Cuda 12.6 next.
UPDATE2:
Downgrading to Cuda 12.6 did not help. It still hangs.
@qiyuxinlin commented on GitHub (Apr 7, 2025):
Hi @createthis, could you please post the files under model_path(deepseek-ai/DeepSeek-V3)?
@createthis commented on GitHub (Apr 7, 2025):
@datouxia I just realized I misread your message. Apologies. You were correct. It's working now!
@NUSTHR commented on GitHub (Apr 7, 2025):
It works! amazing
@hemanthpai commented on GitHub (May 10, 2025):
I'm running ktransformers v0.3 and trying to run the Qwen3-235B-A22B model. I had the same issue where starting server.py would hang at Getting inference context from sched_client.
Downloading all the non-safetensor files from Qwen/Qwen3-235B-A22B into a folder and specifying that folder for --model-path fixed the issue.