[GH-ISSUE #710] RuntimeError: shape '[21, 64]' is invalid for input of size 21 #519

New issue

Closed

opened 2026-10-29 14:40:45 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 14:40:45 +00:00

Owner

Copy link

Originally created by @alkalimc on GitHub (Feb 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/710

Based on the latest code build, the startup parameters are:

/home/test/anaconda3/envs/ktransformers/bin/python /Services/KTransformers/ktransformers/ktransformers/server/main.py \
    --model_path /Repositories/Models/Qwen2-57B-A14B-Instruct/Qwen2-57B-A14B-Instruct \
    --optimize_config_path /Services/KTransformers/ktransformers/ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml \
    --gguf_path /Repositories/Models/Qwen2-57B-A14B-Instruct/Qwen2-57B-A14B-Instruct-GGUF/qwen2-57b-a14b-instruct-q4_0.gguf \
    --max_new_tokens 65536 \
    --cpu_infer 16 \
    --model_name Qwen2-57B-A14B-Instruct-Q4_0 \
    --port 12345

Error message:

2025-02-27 09:30:33,207 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager
INFO:     Started server process [1145]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:12345 (Press CTRL+C to quit)
INFO:     127.0.0.1:45066 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:44262 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:44278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-02-27 09:44:33,240 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 21])
2025-02-27 09:44:33,242 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[133]: input_ids: torch.Size([1, 21])
2025-02-27 09:44:33,242 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[159]: same prefix len: 0
2025-02-27 09:44:33,244 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[168]: input_ids: torch.Size([1, 21])
2025-02-27 09:44:33,245 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[170]: generate_ids: torch.Size([1, 0])
2025-02-27 09:44:33,296 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[181]: cache position: 0 to 21
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
    await self.message_event.wait()
  File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/asyncio/locks.py", line 212, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f6616673530

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    |     async for event in async_events:
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    |     async for event in async_events:
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    |     async for event in async_events:
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
    |     async for event in async_events:
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 31, in inner
    |     async for token in interface.inference(input_message,id):
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in inference
    |     async for v in super().inference(local_messages, thread_id):
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 389, in inference
    |     for t in self.prefill(input_ids, self.check_is_new(thread_id)):
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    |     response = gen.send(None)
    |                ^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 193, in prefill
    |     logits = self.model(
    |              ^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 1414, in forward
    |     outputs = self.model(
    |               ^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/operators/models.py", line 362, in forward
    |     layer_outputs = decoder_layer(
    |                     ^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 936, in forward
    |     hidden_states = self.mlp(hidden_states)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/operators/experts.py", line 647, in forward
    |     F.sigmoid(self.shared_expert_gate(hidden_states)) * shared_expert_output
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/operators/linear.py", line 508, in forward
    |     y = self.generate_linear.forward(x)
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Services/KTransformers/ktransformers/ktransformers/operators/linear.py", line 351, in forward
    |     return x.reshape(orig_shape).to(orig_dtype)
    |            ^^^^^^^^^^^^^^^^^^^^^
    | RuntimeError: shape '[21, 64]' is invalid for input of size 21
    +------------------------------------

Originally created by @alkalimc on GitHub (Feb 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/710 Based on the latest code build, the startup parameters are: ``` /home/test/anaconda3/envs/ktransformers/bin/python /Services/KTransformers/ktransformers/ktransformers/server/main.py \ --model_path /Repositories/Models/Qwen2-57B-A14B-Instruct/Qwen2-57B-A14B-Instruct \ --optimize_config_path /Services/KTransformers/ktransformers/ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml \ --gguf_path /Repositories/Models/Qwen2-57B-A14B-Instruct/Qwen2-57B-A14B-Instruct-GGUF/qwen2-57b-a14b-instruct-q4_0.gguf \ --max_new_tokens 65536 \ --cpu_infer 16 \ --model_name Qwen2-57B-A14B-Instruct-Q4_0 \ --port 12345 ``` Error message: ``` 2025-02-27 09:30:33,207 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager INFO: Started server process [1145] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:12345 (Press CTRL+C to quit) INFO: 127.0.0.1:45066 - "GET /v1/models HTTP/1.1" 200 OK INFO: 127.0.0.1:44262 - "GET /v1/models HTTP/1.1" 200 OK INFO: 127.0.0.1:44278 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2025-02-27 09:44:33,240 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 21]) 2025-02-27 09:44:33,242 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[133]: input_ids: torch.Size([1, 21]) 2025-02-27 09:44:33,242 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[159]: same prefix len: 0 2025-02-27 09:44:33,244 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[168]: input_ids: torch.Size([1, 21]) 2025-02-27 09:44:33,245 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[170]: generate_ids: torch.Size([1, 0]) 2025-02-27 09:44:33,296 DEBUG /Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[181]: cache position: 0 to 21 ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 268, in __call__ await wrap(partial(self.listen_for_disconnect, receive)) File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 264, in wrap await func() File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 233, in listen_for_disconnect message = await receive() ^^^^^^^^^^^^^^^ File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive await self.message_event.wait() File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/asyncio/locks.py", line 212, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f6616673530 During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 261, in __call__ | async with anyio.create_task_group() as task_group: | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 264, in wrap | await func() | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/starlette/responses.py", line 245, in stream_response | async for chunk in self.body_iterator: | File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link | async for event in async_events: | File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply | async for event in async_events: | File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done | async for event in async_events: | File "/Services/KTransformers/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk | async for event in async_events: | File "/Services/KTransformers/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 31, in inner | async for token in interface.inference(input_message,id): | File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in inference | async for v in super().inference(local_messages, thread_id): | File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 389, in inference | for t in self.prefill(input_ids, self.check_is_new(thread_id)): | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context | response = gen.send(None) | ^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 193, in prefill | logits = self.model( | ^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 1414, in forward | outputs = self.model( | ^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/operators/models.py", line 362, in forward | layer_outputs = decoder_layer( | ^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 936, in forward | hidden_states = self.mlp(hidden_states) | ^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/operators/experts.py", line 647, in forward | F.sigmoid(self.shared_expert_gate(hidden_states)) * shared_expert_output | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/test/anaconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/operators/linear.py", line 508, in forward | y = self.generate_linear.forward(x) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Services/KTransformers/ktransformers/ktransformers/operators/linear.py", line 351, in forward | return x.reshape(orig_shape).to(orig_dtype) | ^^^^^^^^^^^^^^^^^^^^^ | RuntimeError: shape '[21, 64]' is invalid for input of size 21 +------------------------------------ ```

bsakaguchi closed this issue 2026-10-29 14:40:45 +00:00

bsakaguchi commented 2026-10-29 14:40:46 +00:00

Author

Owner

Copy link

@yunxi177 commented on GitHub (Feb 27, 2025):

the same

<!-- gh-comment-id:2686697363 --> @yunxi177 commented on GitHub (Feb 27, 2025): the same

bsakaguchi commented 2026-10-29 14:40:46 +00:00

Author

Owner

Copy link

@alkalimc commented on GitHub (Mar 4, 2025):

The issue has been fixed in the latest source code build, and this issue is now closed.

<!-- gh-comment-id:2695851065 --> @alkalimc commented on GitHub (Mar 4, 2025): The issue has been fixed in the latest source code build, and this issue is now closed.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#519

No description provided.