mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #914] [Bug] RuntimeError: CUDA error: invalid device function #653
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#653
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @a11031031 on GitHub (Mar 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/914
检查清单
问题描述
通过
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./deepseek-ai/DeepSeek-V2-Lite-Chat-GGUF命令可以正常运行 但是输出异常Chat: 3.8大还是3.11 大
3大还是3.11 大
User: 3大还是3.11 大
Assistant: 3大还是3.11 大
....
一直循环
通过
ktransformers --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./deepseek-ai/DeepSeek-V2-Lite-Chat-GGUF --port 10002命令运行 可以正常运行，通过命令调用接口时报错curl -X 'POST' 'http://localhost:10002/api/generate' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ "model": "deepseek_r1", "prompt": "tell me a joke.", "stream": true }' curl: (18) transfer closed with outstanding read data remaining`COMPLETION INPUT:----
tell me a joke.
INFO: 127.0.0.1:50824 - "POST /api/generate HTTP/1.1" 200 OK
2025-03-17 16:56:28,229 DEBUG /root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 6])
2025-03-17 16:56:28,230 DEBUG /root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 0
2025-03-17 16:56:28,233 DEBUG /root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 6])
2025-03-17 16:56:28,233 DEBUG /root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 0])
2025-03-17 16:56:28,235 DEBUG /root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 0 to 6
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 270, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 234, in listen_for_disconnect
message = await receive()
^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
await self.message_event.wait()
File "/root/miniconda/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 75c185706d50
During handling of the above exception, another exception occurred:
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 263, in call
| async with anyio.create_task_group() as task_group:
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
| await self.app(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in call
| await self.middleware_stack(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in call
| with collapse_excgroups():
| File "/root/miniconda/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in exit
| self.gen.throw(typ, value, traceback)
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/ollama/completions.py", line 63, in inner
| async for res in interface.inference(input.prompt, id):
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p):
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 403, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p):
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 200, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 729, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward
| hidden_states, self_attn_weights, present_key_value = self.self_attn(
| ^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 599, in forward
| return self.forward_windows(
| ^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 539, in forward_windows
| return self.forward_chunck(
| ^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 98, in forward_chunck
| compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 677, in forward
| y = self.generate_linear.forward(x)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 495, in forward
| padding_input[:,:self.orin_in_features] = x
| ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
| RuntimeError: CUDA error: invalid device function
| Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.|
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in call
with collapse_excgroups():
File "/root/miniconda/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in exit
self.gen.throw(typ, value, traceback)
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/ollama/completions.py", line 63, in inner
async for res in interface.inference(input.prompt, id):
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p):
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 403, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p):
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 200, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
outputs = self.model(
^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 729, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 599, in forward
return self.forward_windows(
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 539, in forward_windows
return self.forward_chunck(
^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 98, in forward_chunck
compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 677, in forward
y = self.generate_linear.forward(x)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 495, in forward
padding_input[:,:self.orin_in_features] = x
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.`复现步骤
ktransformers --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./deepseek-ai/DeepSeek-V2-Lite-Chat-GGUF --port 10002
环境信息
配置和环境如下
ktransformers 0.2.3.post2+torch26fancy
torch 2.6.0+cu124
torchaudio 2.6.0+cu124
torchvision 0.21.0+cu124
`nvidia-smi
Mon Mar 17 16:58:43 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1080 Off | 00000000:3B:00.0 Off | N/A |
| 19% 28C P0 39W / 180W | 0MiB / 8192MiB | 1% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce GTX 1080 Off | 00000000:AF:00.0 Off | N/A |
| 19% 24C P0 35W / 180W | 0MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
`
@a11031031 commented on GitHub (Mar 17, 2025):
内存 128G
CPU Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz * 2
显卡 NV 1080 * 2
@a11031031 commented on GitHub (Mar 18, 2025):
python ktransformers/server/main.py --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./deepseek-ai/DeepSeek-V2-Lite-Chat-GGUF --cpu_infer 22 --max_new_tokens 1000 --host 0.0.0.0 --port 10002 --model_name deepseek_v2通过此命令可以正常运行 也可以正常调用API聊天。但是响应内容有点问题？？