mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #718] [BUG] Multiple GPUs report errors during conversations #526
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#526
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @charliboy on GitHub (Feb 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/718
Describe the bug
A clear and concise description of what the bug is.
/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py:105: FutureWarning:
torch.backends.cuda.sdp_kernel()is deprecated. In the future, this context manager will be removed. Please seetorch.nn.attention.sdpa_kernel()for the new context manager, with updated signature.self.gen = func(*args, **kwds)
2025-02-27 11:27:04,751 - INFO - flashinfer.jit: Loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64
/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-02-27 11:27:04,809 - INFO - flashinfer.jit: Finished loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64
/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py:105: FutureWarning:
torch.backends.cuda.sdp_kernel()is deprecated. In the future, this context manager will be removed. Please seetorch.nn.attention.sdpa_kernel()for the new context manager, with updated signature.self.gen = func(*args, **kwds)
CUDA Error: invalid argument (1) /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include/flashinfer/attention/scheduler.cuh: line 1248 at function cudaMemcpyAsync(int_buffer, page_locked_int_buffer, num_bytes_to_copy, cudaMemcpyHostToDevice, stream)
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 268, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
await func()
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
message = await receive()
^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
await self.message_event.wait()
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7eecac7f3e50
During handling of the above exception, another exception occurred:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
| await self.app(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 715, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
| await route.handle(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 261, in call
| async with anyio.create_task_group() as task_group:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
| await func()
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
| async for chunk in self.body_iterator:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 31, in inner
| async for token in interface.inference(input_message,id):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in inference
| async for v in super().inference(local_messages, thread_id):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 397, in inference
| for t in self.generate():
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 57, in generator_context
| response = gen.send(request)
| ^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 337, in generate
| MLAWrapperSingleton.plan_all(None,None,None,self.active_cache_position.to(torch.int32)+1,
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/flashinfer_wrapper.py", line 171, in plan_all
| wrapper.plan(qo_indptr,
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/flashinfer_wrapper.py", line 126, in plan
| self.wrapper.plan(
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/mla.py", line 251, in plan
| self._plan_info = self._cached_module.plan(
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/_ops.py", line 1123, in call
| return self._op(*args, **(kwargs or {}))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_device.py", line 104, in torch_function
| return func(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/_ops.py", line 1123, in call
| return self._op(*args, **(kwargs or {}))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| RuntimeError: Failed to plan MLA, error: invalid argument
To Reproduce
Steps to reproduce the behavior:
export CUDA_VISIBLE_DEVICES=0,1,2
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path ../DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL --model_name DeepSeek-R1-Q2:671B --max_new_tokens 16384 --cpu_infer 32 --temperature 0.6 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-R1-Chat-Muti-gpu.yaml --force_think --port 10001
Screenshots
Can load normally, but reports an error when starting a conversation
Environments(please complete the following information):
Ubuntu 22.04， NVIDIA RTX A6000 ada，ktransformer 0.2.2rc
Additional context
Add any other context about the problem here.
@AGenchev commented on GitHub (Mar 5, 2025):
I get the same error with multiple gpus template. Works with single GPU though.
@juanqui commented on GitHub (Mar 10, 2025):
Built
v0.2.3post1from source, same issue. Works with 1 GPU, not 2.