kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #70] UnboundLocalError: cannot access local variable 'chunck_mask' where it is not associated with a value #36

New issue
Closed
opened 2026-10-29 14:32:52 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-29 14:32:52 +00:00
Owner
Copy link

Originally created by @fengyang95 on GitHub (Aug 31, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/70

configuration: /DeepSeek-V2-Chat-multi-gpu-4.yaml
When the input prompt is relatively long, the following error appears：

2024-09-01 01:52:15,535 DEBUG /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 1982])
2024-09-01 01:52:15,536 DEBUG /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 1982
INFO:     127.0.0.1:59126 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 754, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 774, in app
    await route.handle(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 295, in handle
    await self.app(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 297, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 210, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion
    async for token in interface.inference(input_message,id):
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference
    for t in self.prefill(input_ids,self.check_is_new(thread_id)):
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill
    logits = self.model(
             ^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/models.py", line 719, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 202, in forward
    chunck_mask,
    ^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'chunck_mask' where it is not associated with a value
Originally created by @fengyang95 on GitHub (Aug 31, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/70 configuration: /DeepSeek-V2-Chat-multi-gpu-4.yaml When the input prompt is relatively long, the following error appears： ```bash 2024-09-01 01:52:15,535 DEBUG /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 1982]) 2024-09-01 01:52:15,536 DEBUG /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 1982 INFO: 127.0.0.1:59126 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 754, in __call__ await self.middleware_stack(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 774, in app await route.handle(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 295, in handle await self.app(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/starlette/routing.py", line 74, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 297, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 210, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion async for token in interface.inference(input_message,id): File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference for t in self.prefill(input_ids,self.check_is_new(thread_id)): File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill logits = self.model( ^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/models.py", line 719, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 202, in forward chunck_mask, ^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'chunck_mask' where it is not associated with a value ```
bsakaguchi commented 2026-10-29 14:32:53 +00:00
Author
Owner
Copy link

@sayap commented on GitHub (Sep 2, 2024):

Need to change chunk_mask in line 186 to chunck_mask (e.g. see github.com/sayap/ktransformers@b38e49220d)

<!-- gh-comment-id:2323715247 --> @sayap commented on GitHub (Sep 2, 2024): Need to change `chunk_mask` in line 186 to `chunck_mask` (e.g. see https://github.com/sayap/ktransformers/commit/b38e49220d)
bsakaguchi commented 2026-10-29 14:32:53 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Sep 2, 2024):

It seems caused by my typo orz, Fixed by #71

<!-- gh-comment-id:2323757225 --> @Azure-Tang commented on GitHub (Sep 2, 2024): It seems caused by my typo orz, Fixed by #71
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#36
No description provided.