kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #171] 1T内存混合4*3090推理问题 #95

New issue
Closed
opened 2026-10-29 14:34:08 +00:00 by bsakaguchi · 10 comments
bsakaguchi commented 2026-10-29 14:34:08 +00:00
Owner
Copy link

Originally created by @edward-ng-duke on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/171

情况
目前使用的是ktransformers0.2.0, 使用配置为 4*3090 + 1T DDR4 RAM, 按照whl安装项目

启动命令
启动命令为ktransformers --model_path ./DeepSeek-R1 --gguf_path /home/weiqu/r1_full_model/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/ --port 5000 --host 0.0.0.0 --cpu_infer 25 --optimize_config_path /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --cache_lens 12000

bug情况
已经显示后台启动成功:
INFO: Started server process [4573]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO: 10.0.0.64:52317 - "POST /v1/chat/completions HTTP/1.1" 200 OK

但是随即报错,以下是完整log
oading blk.60.attn_kv_a_norm.weight to cuda:1
loading blk.60.attn_kv_b.weight to cuda:1
loading blk.60.attn_norm.weight to cuda:1
loading blk.60.ffn_norm.weight to cuda:1
loading output_norm.weight to cuda:1
loading output.weight to cuda:1
2025-02-12 17:44:14,077 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager
INFO: Started server process [4573]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO: 10.0.0.64:52317 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-02-12 17:44:31,847 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 44])
2025-02-12 17:44:31,847 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[122]: input_ids: torch.Size([1, 44])
2025-02-12 17:44:31,868 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[148]: cache position: 0 to 44
/home/weiqu/miniconda3/envs/k021/lib/python3.11/contextlib.py:105: FutureWarning: torch.backends.cuda.sdp_kernel() is deprecated. In the future, this context manager will be removed. Please see torch.nn.attention.sdpa_kernel() for the new context manager, with updated signature.
self.gen = func(*args, **kwds)
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 268, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
await func()
File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
message = await receive()
^^^^^^^^^^^^^^^
File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
await self.message_event.wait()
File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7fab542f29d0

During handling of the above exception, another exception occurred:

  • Exception Group Traceback (most recent call last):
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    | result = await app( # type: ignore[func-returns-value]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
    | return await self.app(scope, receive, send)
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
    | await super().call(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/applications.py", line 112, in call
    | await self.middleware_stack(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
    | raise exc
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
    | await self.app(scope, receive, _send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
    | await self.app(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
    | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    | raise exc
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    | await app(scope, receive, sender)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 715, in call
    | await self.middleware_stack(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    | await route.handle(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    | await self.app(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    | raise exc
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    | await app(scope, receive, sender)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    | await response(scope, receive, send)
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 261, in call
    | async with anyio.create_task_group() as task_group:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in aexit
    | raise BaseExceptionGroup(
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    | await func()
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    | async for chunk in self.body_iterator:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    | async for event in async_events:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    | async for event in async_events:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    | async for event in async_events:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
    | async for event in async_events:
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 26, in inner
    | async for token in interface.inference(input_message,id):
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 337, in inference
    | for t in self.generate():
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    | response = gen.send(None)
    | ^^^^^^^^^^^^^^
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 295, in generate
    | next_token = self.decode_one_tokens()
    | ^^^^^^^^^^^^^^^^^^^^^^^^
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 98, in decode_one_tokens
    | return self.logits_to_token(logits)
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 213, in logits_to_token
    | last = torch.multinomial(probs, num_samples=1)
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | RuntimeError: CUDA error: device-side assert triggered
    | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
    | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
    | Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
    |
    +------------------------------------
    /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed.
    ^CINFO: Shutting down
    INFO: Waiting for application shutdown.
    INFO: Application shutdown complete.
    INFO: Finished server process [4573]
    Aborted (core dumped)
Originally created by @edward-ng-duke on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/171 **情况** 目前使用的是ktransformers0.2.0, 使用配置为 4*3090 + 1T DDR4 RAM, 按照whl安装项目 **启动命令** 启动命令为ktransformers --model_path ./DeepSeek-R1 --gguf_path /home/weiqu/r1_full_model/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/ --port 5000 --host 0.0.0.0 --cpu_infer 25 --optimize_config_path /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --cache_lens 12000 **bug情况** 已经显示后台启动成功: INFO: Started server process [4573] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit) INFO: 10.0.0.64:52317 - "POST /v1/chat/completions HTTP/1.1" 200 OK **但是随即报错,以下是完整log** oading blk.60.attn_kv_a_norm.weight to cuda:1 loading blk.60.attn_kv_b.weight to cuda:1 loading blk.60.attn_norm.weight to cuda:1 loading blk.60.ffn_norm.weight to cuda:1 loading output_norm.weight to cuda:1 loading output.weight to cuda:1 2025-02-12 17:44:14,077 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager INFO: Started server process [4573] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit) INFO: 10.0.0.64:52317 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2025-02-12 17:44:31,847 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 44]) 2025-02-12 17:44:31,847 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[122]: input_ids: torch.Size([1, 44]) 2025-02-12 17:44:31,868 DEBUG /home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[148]: cache position: 0 to 44 /home/weiqu/miniconda3/envs/k021/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature. self.gen = func(*args, **kwds) ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__ await wrap(partial(self.listen_for_disconnect, receive)) File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap await func() File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect message = await receive() ^^^^^^^^^^^^^^^ File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive await self.message_event.wait() File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/asyncio/locks.py", line 213, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope 7fab542f29d0 During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap | await func() | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response | async for chunk in self.body_iterator: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link | async for event in async_events: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply | async for event in async_events: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done | async for event in async_events: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk | async for event in async_events: | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 26, in inner | async for token in interface.inference(input_message,id): | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 337, in inference | for t in self.generate(): | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context | response = gen.send(None) | ^^^^^^^^^^^^^^ | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 295, in generate | next_token = self.decode_one_tokens() | ^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 98, in decode_one_tokens | return self.logits_to_token(logits) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/weiqu/miniconda3/envs/k021/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 213, in logits_to_token | last = torch.multinomial(probs, num_samples=1) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | RuntimeError: CUDA error: device-side assert triggered | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. | For debugging consider passing CUDA_LAUNCH_BLOCKING=1 | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. | +------------------------------------ /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed. ^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4573] Aborted (core dumped)
bsakaguchi commented 2026-10-29 14:34:08 +00:00
Author
Owner
Copy link

@edward-ng-duke commented on GitHub (Feb 12, 2025):

补充
这个情况只在DeepSeek-R1-UD-IQ1_S中出现, 也就是unsloth的1.58bit动态量化版本
把模型换成DeepSeek-R1-UD-Q2_K_XL就好了, 综合bug info , 是不是量化位数太低导致torch的推理问题?

<!-- gh-comment-id:2653257231 --> @edward-ng-duke commented on GitHub (Feb 12, 2025): **补充** 这个情况只在DeepSeek-R1-UD-IQ1_S中出现, 也就是unsloth的1.58bit动态量化版本 把模型换成DeepSeek-R1-UD-Q2_K_XL就好了, 综合bug info , 是不是量化位数太低导致torch的推理问题?
bsakaguchi commented 2026-10-29 14:34:08 +00:00
Author
Owner
Copy link

@20000419 commented on GitHub (Feb 12, 2025):

请问如何配置多GPU？您是让4张3090都参与运算了吗？

<!-- gh-comment-id:2653531335 --> @20000419 commented on GitHub (Feb 12, 2025): 请问如何配置多GPU？您是让4张3090都参与运算了吗？
bsakaguchi commented 2026-10-29 14:34:09 +00:00
Author
Owner
Copy link

@Kiciot commented on GitHub (Feb 12, 2025):

请问如何配置多GPU？您是让4张3090都参与运算了吗？

配置多GPU网上很多，可以找一下，一般来说四到八块能并行运算。不过更多是为了拓展显存

<!-- gh-comment-id:2653555141 --> @Kiciot commented on GitHub (Feb 12, 2025): > 请问如何配置多GPU？您是让4张3090都参与运算了吗？ 配置多GPU网上很多，可以找一下，一般来说四到八块能并行运算。不过更多是为了拓展显存
bsakaguchi commented 2026-10-29 14:34:09 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Feb 12, 2025):

补充 这个情况只在DeepSeek-R1-UD-IQ1_S中出现, 也就是unsloth的1.58bit动态量化版本 把模型换成DeepSeek-R1-UD-Q2_K_XL就好了, 综合bug info , 是不是量化位数太低导致torch的推理问题?

是目前支持了Qx_k和IQ4，你先试试它们。后续我们会加上这些支持的~

<!-- gh-comment-id:2653582013 --> @KMSorSMS commented on GitHub (Feb 12, 2025): > **补充** 这个情况只在DeepSeek-R1-UD-IQ1_S中出现, 也就是unsloth的1.58bit动态量化版本 把模型换成DeepSeek-R1-UD-Q2_K_XL就好了, 综合bug info , 是不是量化位数太低导致torch的推理问题? 是目前支持了Qx_k和IQ4，你先试试它们。后续我们会加上这些支持的~
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@aimms321 commented on GitHub (Feb 12, 2025):

楼主能测一下128K的上下文对显存的占用量么？谢谢 @edward-ng-duke

<!-- gh-comment-id:2653614398 --> @aimms321 commented on GitHub (Feb 12, 2025): 楼主能测一下128K的上下文对显存的占用量么？谢谢 @edward-ng-duke
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@zerotorunaway commented on GitHub (Feb 12, 2025):

4卡盟提升推理速度吗？在这个框架下？能提升多少

<!-- gh-comment-id:2653773350 --> @zerotorunaway commented on GitHub (Feb 12, 2025): 4卡盟提升推理速度吗？在这个框架下？能提升多少
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@edward-ng-duke commented on GitHub (Feb 13, 2025):

4卡盟提升推理速度吗？在这个框架下？能提升多少

文档中提到1卡和2卡速度区别不大, 只有上下文长度的区别, 这和我的实际测试是一致的

<!-- gh-comment-id:2655123183 --> @edward-ng-duke commented on GitHub (Feb 13, 2025): > 4卡盟提升推理速度吗？在这个框架下？能提升多少 文档中提到1卡和2卡速度区别不大, 只有上下文长度的区别, 这和我的实际测试是一致的
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@edward-ng-duke commented on GitHub (Feb 13, 2025):

请问如何配置多GPU？您是让4张3090都参与运算了吗？

直接使用multi_gpu.yaml即可, 多个GPU按我的理解只提升上下文

<!-- gh-comment-id:2655125949 --> @edward-ng-duke commented on GitHub (Feb 13, 2025): > 请问如何配置多GPU？您是让4张3090都参与运算了吗？ 直接使用multi_gpu.yaml即可, 多个GPU按我的理解只提升上下文
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@magicyuan876 commented on GitHub (Feb 13, 2025):

请问下4卡的上下文最大能支持到多少？并发性能怎样？

<!-- gh-comment-id:2655182796 --> @magicyuan876 commented on GitHub (Feb 13, 2025): 请问下4卡的上下文最大能支持到多少？并发性能怎样？
bsakaguchi commented 2026-10-29 14:34:10 +00:00
Author
Owner
Copy link

@ruralharry commented on GitHub (Feb 13, 2025):

请问下4卡的上下文最大能支持到多少？并发性能怎样？
同问，4卡到底有哪些优势和劣势啊

<!-- gh-comment-id:2655700708 --> @ruralharry commented on GitHub (Feb 13, 2025): 请问下4卡的上下文最大能支持到多少？并发性能怎样？ 同问，4卡到底有哪些优势和劣势啊
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#95
No description provided.