kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1570] [Bug] Shared memory crash running Kimi K2 Thinking with sglang on 4x RTX 6000 Workstation Pro #1038

New issue
Closed
opened 2026-10-29 14:47:42 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-29 14:47:42 +00:00
Owner
Copy link

Originally created by @0xhaggis on GitHub (Nov 8, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1570

Originally assigned to: @Atream on GitHub.

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Quick description

Following the example at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2-Thinking.md there is a repeatable crash running sglang with the ktransformers CPU kernel. The error message is:

triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or num_stages may help.

This crash under 2 known conditions:

  1. The second request sent to sglang's HTTP server after startup will always cause the crash; the first request after startup is always answered as expected, unless:
  2. The incoming request is too large (i.e. contains too many tokens for available resources). Then crash.

Reproduction

Side note

The arg --kt-threadpool-count 2 must always be 2 otherwise Bad Things happen. For example, setting it to 1 causes the model to output a never-ending stream of only the lowercase letter "b".

Example #1: crash upon second request

  1. The sglang/ktransformer server is launched
  2. A first request is made to the LLM ("write flappy bird")
  3. The LLM responds with a working flappy bird game
  4. Make another request of the LLM for anything. "Say hello".
  5. Crash
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8080 \
--model moonshotai/Kimi-K2-Thinking \
--kt-amx-weight-path ~/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/ \
--kt-cpuinfer 126 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 224 \
--kt-amx-method AMXINT4 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.985 \
--chunked-prefill-size 4096 \
--max-running-requests 1 \
--max-total-tokens 131072 \
--enable-mixed-chunk \
--tensor-parallel-size 4 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--served-model-name kimi
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 16:30:44] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 16:30:45] server_args=ServerArgs(model_path='moonshotai/Kimi-K2-Thinking', tokenizer_path='moonshotai/Kimi-K2-Thinking', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8080, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.985, max_running_requests=1, max_queued_requests=None, max_total_tokens=131072, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=4, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=865438033, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', api_key=None, served_model_name='kimi', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_amx_weight_path='/home/el_duderino/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/', kt_amx_method='AMXINT4', kt_cpuinfer=126, kt_threadpool_count=2, kt_num_gpu_experts=224, kt_max_deferred_experts_per_token=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=512, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, decrypted_config_file=None, decrypted_draft_config_file=None)
[2025-11-07 16:30:45] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:45] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:30:45] Using default HuggingFace chat template with detected content format: openai
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
[2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 16:30:50 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:50 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 16:30:51] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:51] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:30:51 TP1] Init torch distributed begin.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:30:51 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:51 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 16:30:51 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:51 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 16:30:51 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:30:51 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:30:51 TP0] Init torch distributed begin.
[2025-11-07 16:30:51 TP3] Init torch distributed begin.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:30:51 TP2] Init torch distributed begin.
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[2025-11-07 16:30:52 TP0] sglang is using nccl==2.27.3
[2025-11-07 16:30:52 TP3] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 16:30:52 TP1] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 16:30:52 TP0] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 16:30:52 TP2] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[2025-11-07 16:30:52 TP0] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 16:30:52 TP3] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 16:30:52 TP2] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 16:30:52 TP1] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 16:30:53 TP1] Load weight begin. avail mem=94.07 GB
[2025-11-07 16:30:53 TP3] Load weight begin. avail mem=94.07 GB
[2025-11-07 16:30:53 TP2] Load weight begin. avail mem=94.07 GB
[2025-11-07 16:30:53 TP0] Load weight begin. avail mem=94.07 GB
[2025-11-07 16:30:53 TP0] Detected fp8 checkpoint.
CPUInfer[0x55597315e120]: Hello
WorkerPool[0x55597315d250] 2 subpools, [numa:threads][0:63] [1:63]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 63 threads
numa_sched_setaffinity_v2_int() failed: Invalid argument
set_mempolicy: Invalid argument
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 63 threads
NUMA node 1 not found
... removed ...
NUMA node 1 not found.
[2025-11-07 16:30:55 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/62 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   2% Completed | 1/62 [00:00<00:51,  1.19it/s]
Loading safetensors checkpoint shards:   3% Completed | 2/62 [00:03<02:08,  2.15s/it]
... removed ...
Loading safetensors checkpoint shards: 100% Completed | 62/62 [02:57<00:00,  2.86s/it]

[2025-11-07 16:33:55 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 16:33:55 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 16:33:55 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
TP MOE layer 1, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 1 at numa 0
Creating AMX_MOE_TP 0 at numa 0
alloc 1 from other numa for 75db58000b70
debug physical_to_logical_map in arg:93842517923200
moe ptr:0x55597fb88ca0,confirm: moe->config.physical_to_logical_map:93842517923200
... removed ...
TP Load from loader
TP MOE layer 59, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 74b51ff36490
debug physical_to_logical_map in arg:93842518101376
moe ptr:0x55597a68bb90,confirm: moe->config.physical_to_logical_map:93842518101376
TP Load from loader
TP MOE layer 60, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 74b337f36820
debug physical_to_logical_map in arg:93842518104448
moe ptr:0x555979cde100,confirm: moe->config.physical_to_logical_map:93842518104448
TP Load from loader
[2025-11-07 16:37:36 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 16:37:36 TP0] Using KV cache dtype: torch.bfloat16
[2025-11-07 16:37:36 TP0] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 16:37:36 TP3] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 16:37:36 TP2] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 16:37:36 TP1] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 16:37:36 TP3] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 16:37:36 TP0] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 16:37:36 TP3] Memory pool end. avail mem=1.52 GB
[2025-11-07 16:37:36 TP0] Memory pool end. avail mem=1.52 GB
[2025-11-07 16:37:36 TP2] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 16:37:36 TP2] Memory pool end. avail mem=1.52 GB
[2025-11-07 16:37:36 TP1] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 16:37:36 TP1] Memory pool end. avail mem=1.52 GB
[2025-11-07 16:37:36 TP3] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 16:37:36 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 16:37:36 TP0] Capture cuda graph bs [1]
[2025-11-07 16:37:36 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 16:37:36 TP2] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
Capturing batches (bs=1 avail_mem=1.51 GB):   0%|                                                                                                                     | 0/1 [00:00<?, ?it/s][2025-11-07 16:37:37 TP3] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 16:37:37 TP1] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 16:37:37 TP2] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 16:37:37 TP0] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
Capturing batches (bs=1 avail_mem=1.51 GB): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.03s/it]
[2025-11-07 16:37:38 TP3] Capture cuda graph end. Time elapsed: 1.60 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 16:37:38 TP2] Capture cuda graph end. Time elapsed: 1.60 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 16:37:38 TP0] Capture cuda graph end. Time elapsed: 1.62 s. mem usage=0.05 GB. avail mem=1.46 GB.
[2025-11-07 16:37:38 TP1] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 16:37:39 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:37:39 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 16:37:39 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:37:39 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:37:39 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 16:37:39 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 16:37:39 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 16:37:39 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 16:37:39 TP0] max_total_num_tokens=101872, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=1, context_len=262144, available_gpu_mem=1.46 GB
[2025-11-07 16:37:39] INFO:     Started server process [371392]
[2025-11-07 16:37:39] INFO:     Waiting for application startup.
[2025-11-07 16:37:39] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-11-07 16:37:39] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-11-07 16:37:39] INFO:     Application startup complete.
[2025-11-07 16:37:39] INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
[2025-11-07 16:37:40] INFO:     127.0.0.1:42088 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-11-07 16:37:40 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 16:37:41] INFO:     127.0.0.1:42098 - "POST /generate HTTP/1.1" 200 OK
[2025-11-07 16:37:41] The server is fired up and ready to roll!
[2025-11-07 17:32:43] INFO:     127.0.0.1:48368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-11-07 17:32:43 TP0] Prefill batch, #new-seq: 1, #new-token: 37, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 17:32:44 TP0] Decode batch, #running-req: 1, #token: 70, token usage: 0.00, cuda graph: True, gen throughput (token/s): 0.01, #queue-req: 0,
[2025-11-07 17:32:46 TP0] Decode batch, #running-req: 1, #token: 110, token usage: 0.00, cuda graph: True, gen throughput (token/s): 32.49, #queue-req: 0,
[2025-11-07 17:32:47 TP0] Decode batch, #running-req: 1, #token: 150, token usage: 0.00, cuda graph: True, gen throughput (token/s): 32.79, #queue-req: 0,
... removed ...
[2025-11-07 17:41:47 TP0] Decode batch, #running-req: 1, #token: 14670, token usage: 0.14, cuda graph: True, gen throughput (token/s): 23.06, #queue-req: 0,
[2025-11-07 17:48:13] INFO:     127.0.0.1:41664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-11-07 17:48:13 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 36, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 17:48:13 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 17:48:13 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 17:48:13 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 17:48:13 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 17:48:13] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[1]    371392 killed     CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --host 0.0.0.0

Environment

Server details

  • Linux Ubuntu
  • AMD EPYC 9755 CPU
  • 768GB system RAM
  • 4x RTX 6000 PRO Workstation GPUs
  • Nvidia drivers 580.95.05
  • CUDA 13.0
  • Python 3.11 with uv 0.7.20

Software setup

# setup virtual python environment
cd
mkdir ktransformers-sglang; cd ktransformers-sglang
uv venv --python 3.11 --seed
. .venv/bin/activate

# install sglang
uv pip install "sglang" --prerelease=allow

#  download and initialize ktransformers repo
git clone https://github.com/kvcache-ai/ktransformers
git submodule update --init --recursive

# Install ktransformers CPU kernel for sglang
cd kt-kernel
export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF
uv pip install .
cd ..

# download Kimi K2 Thinking GPU & CPU parts
uv pip install -U hf hf_transfer
hf download moonshotai/Kimi-K2-Thinking
hf download KVCache-ai/Kimi-K2-Thinking-CPU-weight
Originally created by @0xhaggis on GitHub (Nov 8, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1570 Originally assigned to: @Atream on GitHub. ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug ### Quick description Following the example at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2-Thinking.md there is a repeatable crash running sglang with the ktransformers CPU kernel. The error message is: `triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.` This crash under 2 known conditions: 1. The second request sent to sglang's HTTP server after startup will always cause the crash; the first request after startup is always answered as expected, unless: 2. The incoming request is too large (i.e. contains too many tokens for available resources). Then crash. ### Reproduction ### Side note The arg `--kt-threadpool-count 2` must always be `2` otherwise Bad Things happen. For example, setting it to `1` causes the model to output a never-ending stream of only the lowercase letter "b". ### Example #1: crash upon second request 1. The sglang/ktransformer server is launched 2. A first request is made to the LLM ("write flappy bird") 3. The LLM responds with a working flappy bird game 4. Make another request of the LLM for anything. "Say hello". 5. Crash ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8080 \ --model moonshotai/Kimi-K2-Thinking \ --kt-amx-weight-path ~/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/ \ --kt-cpuinfer 126 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 224 \ --kt-amx-method AMXINT4 \ --attention-backend triton \ --trust-remote-code \ --mem-fraction-static 0.985 \ --chunked-prefill-size 4096 \ --max-running-requests 1 \ --max-total-tokens 131072 \ --enable-mixed-chunk \ --tensor-parallel-size 4 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --served-model-name kimi /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 16:30:44] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 16:30:45] server_args=ServerArgs(model_path='moonshotai/Kimi-K2-Thinking', tokenizer_path='moonshotai/Kimi-K2-Thinking', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8080, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.985, max_running_requests=1, max_queued_requests=None, max_total_tokens=131072, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=4, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=865438033, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', api_key=None, served_model_name='kimi', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_amx_weight_path='/home/el_duderino/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/', kt_amx_method='AMXINT4', kt_cpuinfer=126, kt_threadpool_count=2, kt_num_gpu_experts=224, kt_max_deferred_experts_per_token=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=512, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, decrypted_config_file=None, decrypted_draft_config_file=None) [2025-11-07 16:30:45] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:45] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:30:45] Using default HuggingFace chat template with detected content format: openai /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] [2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 16:30:50] INFO trace.py:52: opentelemetry package is not installed, tracing disabled You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 16:30:50 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:50 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 16:30:51] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:51] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:30:51 TP1] Init torch distributed begin. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:30:51 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:51 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 16:30:51 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:51 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 16:30:51 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:30:51 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:30:51 TP0] Init torch distributed begin. [2025-11-07 16:30:51 TP3] Init torch distributed begin. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:30:51 TP2] Init torch distributed begin. [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [2025-11-07 16:30:52 TP0] sglang is using nccl==2.27.3 [2025-11-07 16:30:52 TP3] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 16:30:52 TP1] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 16:30:52 TP0] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 16:30:52 TP2] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [2025-11-07 16:30:52 TP0] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 16:30:52 TP3] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 16:30:52 TP2] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 16:30:52 TP1] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 16:30:53 TP1] Load weight begin. avail mem=94.07 GB [2025-11-07 16:30:53 TP3] Load weight begin. avail mem=94.07 GB [2025-11-07 16:30:53 TP2] Load weight begin. avail mem=94.07 GB [2025-11-07 16:30:53 TP0] Load weight begin. avail mem=94.07 GB [2025-11-07 16:30:53 TP0] Detected fp8 checkpoint. CPUInfer[0x55597315e120]: Hello WorkerPool[0x55597315d250] 2 subpools, [numa:threads][0:63] [1:63] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 63 threads numa_sched_setaffinity_v2_int() failed: Invalid argument set_mempolicy: Invalid argument ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 63 threads NUMA node 1 not found ... removed ... NUMA node 1 not found. [2025-11-07 16:30:55 TP0] Using model weights format ['*.safetensors'] Loading safetensors checkpoint shards: 0% Completed | 0/62 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 2% Completed | 1/62 [00:00<00:51, 1.19it/s] Loading safetensors checkpoint shards: 3% Completed | 2/62 [00:03<02:08, 2.15s/it] ... removed ... Loading safetensors checkpoint shards: 100% Completed | 62/62 [02:57<00:00, 2.86s/it] [2025-11-07 16:33:55 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 16:33:55 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 16:33:55 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. TP MOE layer 1, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 1 at numa 0 Creating AMX_MOE_TP 0 at numa 0 alloc 1 from other numa for 75db58000b70 debug physical_to_logical_map in arg:93842517923200 moe ptr:0x55597fb88ca0,confirm: moe->config.physical_to_logical_map:93842517923200 ... removed ... TP Load from loader TP MOE layer 59, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 74b51ff36490 debug physical_to_logical_map in arg:93842518101376 moe ptr:0x55597a68bb90,confirm: moe->config.physical_to_logical_map:93842518101376 TP Load from loader TP MOE layer 60, pool: 0x55597315d250, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 74b337f36820 debug physical_to_logical_map in arg:93842518104448 moe ptr:0x555979cde100,confirm: moe->config.physical_to_logical_map:93842518104448 TP Load from loader [2025-11-07 16:37:36 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 16:37:36 TP0] Using KV cache dtype: torch.bfloat16 [2025-11-07 16:37:36 TP0] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 16:37:36 TP3] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 16:37:36 TP2] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 16:37:36 TP1] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 16:37:36 TP3] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 16:37:36 TP0] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 16:37:36 TP3] Memory pool end. avail mem=1.52 GB [2025-11-07 16:37:36 TP0] Memory pool end. avail mem=1.52 GB [2025-11-07 16:37:36 TP2] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 16:37:36 TP2] Memory pool end. avail mem=1.52 GB [2025-11-07 16:37:36 TP1] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 16:37:36 TP1] Memory pool end. avail mem=1.52 GB [2025-11-07 16:37:36 TP3] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 16:37:36 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 16:37:36 TP0] Capture cuda graph bs [1] [2025-11-07 16:37:36 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 16:37:36 TP2] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB Capturing batches (bs=1 avail_mem=1.51 GB): 0%| | 0/1 [00:00<?, ?it/s][2025-11-07 16:37:37 TP3] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 16:37:37 TP1] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 16:37:37 TP2] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 16:37:37 TP0] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton Capturing batches (bs=1 avail_mem=1.51 GB): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.03s/it] [2025-11-07 16:37:38 TP3] Capture cuda graph end. Time elapsed: 1.60 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 16:37:38 TP2] Capture cuda graph end. Time elapsed: 1.60 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 16:37:38 TP0] Capture cuda graph end. Time elapsed: 1.62 s. mem usage=0.05 GB. avail mem=1.46 GB. [2025-11-07 16:37:38 TP1] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 16:37:39 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:37:39 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 16:37:39 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:37:39 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:37:39 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 16:37:39 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 16:37:39 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 16:37:39 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 16:37:39 TP0] max_total_num_tokens=101872, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=1, context_len=262144, available_gpu_mem=1.46 GB [2025-11-07 16:37:39] INFO: Started server process [371392] [2025-11-07 16:37:39] INFO: Waiting for application startup. [2025-11-07 16:37:39] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-11-07 16:37:39] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-11-07 16:37:39] INFO: Application startup complete. [2025-11-07 16:37:39] INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) [2025-11-07 16:37:40] INFO: 127.0.0.1:42088 - "GET /get_model_info HTTP/1.1" 200 OK [2025-11-07 16:37:40 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 16:37:41] INFO: 127.0.0.1:42098 - "POST /generate HTTP/1.1" 200 OK [2025-11-07 16:37:41] The server is fired up and ready to roll! [2025-11-07 17:32:43] INFO: 127.0.0.1:48368 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-11-07 17:32:43 TP0] Prefill batch, #new-seq: 1, #new-token: 37, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 17:32:44 TP0] Decode batch, #running-req: 1, #token: 70, token usage: 0.00, cuda graph: True, gen throughput (token/s): 0.01, #queue-req: 0, [2025-11-07 17:32:46 TP0] Decode batch, #running-req: 1, #token: 110, token usage: 0.00, cuda graph: True, gen throughput (token/s): 32.49, #queue-req: 0, [2025-11-07 17:32:47 TP0] Decode batch, #running-req: 1, #token: 150, token usage: 0.00, cuda graph: True, gen throughput (token/s): 32.79, #queue-req: 0, ... removed ... [2025-11-07 17:41:47 TP0] Decode batch, #running-req: 1, #token: 14670, token usage: 0.14, cuda graph: True, gen throughput (token/s): 23.06, #queue-req: 0, [2025-11-07 17:48:13] INFO: 127.0.0.1:41664 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-11-07 17:48:13 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 36, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 17:48:13 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 17:48:13 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 17:48:13 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 17:48:13 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 17:48:13] SIGQUIT received. signum=None, frame=None. It usually means one child failed. [1] 371392 killed CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --host 0.0.0.0 ``` ### Environment ### Server details - Linux Ubuntu - AMD EPYC 9755 CPU - 768GB system RAM - 4x RTX 6000 PRO Workstation GPUs - Nvidia drivers 580.95.05 - CUDA 13.0 - Python 3.11 with uv 0.7.20 ### Software setup ```bash # setup virtual python environment cd mkdir ktransformers-sglang; cd ktransformers-sglang uv venv --python 3.11 --seed . .venv/bin/activate # install sglang uv pip install "sglang" --prerelease=allow # download and initialize ktransformers repo git clone https://github.com/kvcache-ai/ktransformers git submodule update --init --recursive # Install ktransformers CPU kernel for sglang cd kt-kernel export CPUINFER_CPU_INSTRUCT=AVX512 export CPUINFER_ENABLE_AMX=OFF uv pip install . cd .. # download Kimi K2 Thinking GPU & CPU parts uv pip install -U hf hf_transfer hf download moonshotai/Kimi-K2-Thinking hf download KVCache-ai/Kimi-K2-Thinking-CPU-weight ```
bsakaguchi 2026-10-29 14:47:42 +00:00
  • closed this issue
  • added the
    bug
         label
bsakaguchi commented 2026-10-29 14:47:43 +00:00
Author
Owner
Copy link

@0xhaggis commented on GitHub (Nov 8, 2025):

Example #2: Crash from long prompt

The following example shows:

  1. Load sglang / ktransformers
  2. Make a request with a prompt of just over 15k tokens (not bytes, tokens)
  3. Crash
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8080 \
--model moonshotai/Kimi-K2-Thinking \
--kt-amx-weight-path ~/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/ \
--kt-cpuinfer 126 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 224 \
--kt-amx-method AMXINT4 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.985 \
--chunked-prefill-size 4096 \
--max-running-requests 1 \
--max-total-tokens 131072 \
--enable-mixed-chunk \
--tensor-parallel-size 4 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--served-model-name kimi
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 18:03:29] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 18:03:29] server_args=ServerArgs(model_path='moonshotai/Kimi-K2-Thinking', tokenizer_path='moonshotai/Kimi-K2-Thinking', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8080, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.985, max_running_requests=1, max_queued_requests=None, max_total_tokens=131072, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=4, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=615453135, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', api_key=None, served_model_name='kimi', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_amx_weight_path='/home/el_duderino/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/', kt_amx_method='AMXINT4', kt_cpuinfer=126, kt_threadpool_count=2, kt_num_gpu_experts=224, kt_max_deferred_experts_per_token=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=512, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, decrypted_config_file=None, decrypted_draft_config_file=None)
[2025-11-07 18:03:30] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:30] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:03:30] Using default HuggingFace chat template with detected content format: openai
[2025-11-07 18:03:34] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
[2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 18:03:35 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:35 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
[2025-11-07 18:03:35 TP0] Init torch distributed begin.
[2025-11-07 18:03:35] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:35] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 18:03:35 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:35 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:03:36 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:36 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 18:03:36 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:03:36 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:03:36 TP3] Init torch distributed begin.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:03:36 TP1] Init torch distributed begin.
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:03:36 TP2] Init torch distributed begin.
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[2025-11-07 18:03:36 TP0] sglang is using nccl==2.27.3
[2025-11-07 18:03:36 TP0] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 18:03:36 TP3] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 18:03:36 TP1] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[2025-11-07 18:03:36 TP2] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[2025-11-07 18:03:36 TP0] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 18:03:36 TP3] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 18:03:36 TP1] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 18:03:36 TP2] Init torch distributed ends. mem usage=0.28 GB
[2025-11-07 18:03:38 TP1] Load weight begin. avail mem=94.07 GB
[2025-11-07 18:03:38 TP3] Load weight begin. avail mem=94.07 GB
[2025-11-07 18:03:38 TP2] Load weight begin. avail mem=94.07 GB
[2025-11-07 18:03:38 TP0] Load weight begin. avail mem=94.07 GB
[2025-11-07 18:03:38 TP0] Detected fp8 checkpoint.
CPUInfer[0x637c87d2e7c0]: Hello
WorkerPool[0x637c87d2d8f0] 2 subpools, [numa:threads][0:63] [1:63]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 63 threads
numa_sched_setaffinity_v2_int() failed: Invalid argument
set_mempolicy: Invalid argument
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 63 threads
NUMA node 1 not found
... removed ...
NUMA node 1 not found.
[2025-11-07 18:03:40 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/62 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   2% Completed | 1/62 [00:00<00:44,  1.38it/s]
Loading safetensors checkpoint shards:   3% Completed | 2/62 [00:03<02:13,  2.22s/it]
... removed ...
Loading safetensors checkpoint shards: 100% Completed | 62/62 [02:58<00:00,  2.89s/it]

[2025-11-07 18:06:41 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 18:06:41 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 18:06:41 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
TP MOE layer 1, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 79e5bc000b70
debug physical_to_logical_map in arg:109386352499840
moe ptr:0x637c93857590,confirm: moe->config.physical_to_logical_map:109386352499840
TP Load from loader
TP MOE layer 2, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 79b5cbf36be0
debug physical_to_logical_map in arg:109386352502912
moe ptr:0x637c8dc33d40,confirm: moe->config.physical_to_logical_map:109386352502912
TP Load from loader
TP MOE layer 3, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 79b183f36790
debug physical_to_logical_map in arg:109386352505984
moe ptr:0x637c8ccfc370,confirm: moe->config.physical_to_logical_map:109386352505984
TP Load from loader
TP MOE layer 4, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 79ad3bf36490
debug physical_to_logical_map in arg:109386352509056
moe ptr:0x637c926482c0,confirm: moe->config.physical_to_logical_map:109386352509056
... removed ...
TP Load from loader
TP MOE layer 59, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 78bff3f36490
debug physical_to_logical_map in arg:109386352678016
moe ptr:0x637c93bad1f0,confirm: moe->config.physical_to_logical_map:109386352678016
TP Load from loader
TP MOE layer 60, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 0
alloc 1 from other numa for 78be0bf36820
debug physical_to_logical_map in arg:109386352681088
moe ptr:0x637c901ef580,confirm: moe->config.physical_to_logical_map:109386352681088
TP Load from loader
[2025-11-07 18:10:22 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB.
[2025-11-07 18:10:22 TP0] Using KV cache dtype: torch.bfloat16
[2025-11-07 18:10:22 TP0] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 18:10:22 TP1] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 18:10:22 TP3] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 18:10:22 TP2] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead.
[2025-11-07 18:10:22 TP0] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 18:10:22 TP0] Memory pool end. avail mem=1.52 GB
[2025-11-07 18:10:22 TP1] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 18:10:22 TP1] Memory pool end. avail mem=1.52 GB
[2025-11-07 18:10:22 TP3] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 18:10:22 TP3] Memory pool end. avail mem=1.52 GB
[2025-11-07 18:10:22 TP2] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB
[2025-11-07 18:10:22 TP2] Memory pool end. avail mem=1.52 GB
[2025-11-07 18:10:22 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 18:10:22 TP0] Capture cuda graph bs [1]
[2025-11-07 18:10:22 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 18:10:22 TP3] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
[2025-11-07 18:10:22 TP2] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB
Capturing batches (bs=1 avail_mem=1.51 GB):   0%|                                                                                                                     | 0/1 [00:00<?, ?it/s][2025-11-07 18:10:23 TP3] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 18:10:23 TP1] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 18:10:23 TP2] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-11-07 18:10:23 TP0] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
Capturing batches (bs=1 avail_mem=1.51 GB): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.06s/it]
[2025-11-07 18:10:24 TP1] Capture cuda graph end. Time elapsed: 1.62 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 18:10:24 TP3] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 18:10:24 TP2] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB.
[2025-11-07 18:10:24 TP0] Capture cuda graph end. Time elapsed: 1.65 s. mem usage=0.05 GB. avail mem=1.46 GB.
[2025-11-07 18:10:25 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:10:25 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 18:10:25 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:10:25 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 18:10:25 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:10:25 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
[2025-11-07 18:10:25 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model
[2025-11-07 18:10:25 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
  warnings.warn(
[2025-11-07 18:10:25 TP0] max_total_num_tokens=101872, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=1, context_len=262144, available_gpu_mem=1.46 GB
[2025-11-07 18:10:25] INFO:     Started server process [373587]
[2025-11-07 18:10:25] INFO:     Waiting for application startup.
[2025-11-07 18:10:25] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-11-07 18:10:25] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-11-07 18:10:25] INFO:     Application startup complete.
[2025-11-07 18:10:25] INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
[2025-11-07 18:10:26] INFO:     127.0.0.1:43600 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-11-07 18:10:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 18:10:27] INFO:     127.0.0.1:43614 - "POST /generate HTTP/1.1" 200 OK
[2025-11-07 18:10:27] The server is fired up and ready to roll!
[2025-11-07 18:16:37] INFO:     127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-11-07 18:16:37 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 18:16:56 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.04, #running-req: 0, #queue-req: 0,
[2025-11-07 18:16:56 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 18:16:56 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 18:16:56 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 18:16:56 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core
    return self.forward_absorb_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core
    attn_output = self.attn_mqa(
                  ^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend
    self.extend_attention_fwd(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-11-07 18:16:56] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[1]    373587 killed     CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --host 0.0.0.0
<!-- gh-comment-id:3506097980 --> @0xhaggis commented on GitHub (Nov 8, 2025): ### Example #2: Crash from long prompt The following example shows: 1. Load sglang / ktransformers 2. Make a request with a prompt of just over 15k tokens (not bytes, tokens) 3. Crash ``` CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8080 \ --model moonshotai/Kimi-K2-Thinking \ --kt-amx-weight-path ~/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/ \ --kt-cpuinfer 126 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 224 \ --kt-amx-method AMXINT4 \ --attention-backend triton \ --trust-remote-code \ --mem-fraction-static 0.985 \ --chunked-prefill-size 4096 \ --max-running-requests 1 \ --max-total-tokens 131072 \ --enable-mixed-chunk \ --tensor-parallel-size 4 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --served-model-name kimi /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 18:03:29] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 18:03:29] server_args=ServerArgs(model_path='moonshotai/Kimi-K2-Thinking', tokenizer_path='moonshotai/Kimi-K2-Thinking', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8080, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.985, max_running_requests=1, max_queued_requests=None, max_total_tokens=131072, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=4, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=615453135, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', api_key=None, served_model_name='kimi', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_amx_weight_path='/home/el_duderino/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83/', kt_amx_method='AMXINT4', kt_cpuinfer=126, kt_threadpool_count=2, kt_num_gpu_experts=224, kt_max_deferred_experts_per_token=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=512, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, decrypted_config_file=None, decrypted_draft_config_file=None) [2025-11-07 18:03:30] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:30] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:03:30] Using default HuggingFace chat template with detected content format: openai [2025-11-07 18:03:34] INFO trace.py:52: opentelemetry package is not installed, tracing disabled You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled [2025-11-07 18:03:35] INFO trace.py:52: opentelemetry package is not installed, tracing disabled You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 18:03:35 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:35 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors. [2025-11-07 18:03:35 TP0] Init torch distributed begin. [2025-11-07 18:03:35] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:35] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 18:03:35 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:35 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:03:36 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:36 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 18:03:36 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:03:36 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:03:36 TP3] Init torch distributed begin. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:03:36 TP1] Init torch distributed begin. /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:03:36 TP2] Init torch distributed begin. [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [2025-11-07 18:03:36 TP0] sglang is using nccl==2.27.3 [2025-11-07 18:03:36 TP0] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 18:03:36 TP3] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 18:03:36 TP1] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [2025-11-07 18:03:36 TP2] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [2025-11-07 18:03:36 TP0] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 18:03:36 TP3] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 18:03:36 TP1] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 18:03:36 TP2] Init torch distributed ends. mem usage=0.28 GB [2025-11-07 18:03:38 TP1] Load weight begin. avail mem=94.07 GB [2025-11-07 18:03:38 TP3] Load weight begin. avail mem=94.07 GB [2025-11-07 18:03:38 TP2] Load weight begin. avail mem=94.07 GB [2025-11-07 18:03:38 TP0] Load weight begin. avail mem=94.07 GB [2025-11-07 18:03:38 TP0] Detected fp8 checkpoint. CPUInfer[0x637c87d2e7c0]: Hello WorkerPool[0x637c87d2d8f0] 2 subpools, [numa:threads][0:63] [1:63] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 63 threads numa_sched_setaffinity_v2_int() failed: Invalid argument set_mempolicy: Invalid argument ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 63 threads NUMA node 1 not found ... removed ... NUMA node 1 not found. [2025-11-07 18:03:40 TP0] Using model weights format ['*.safetensors'] Loading safetensors checkpoint shards: 0% Completed | 0/62 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 2% Completed | 1/62 [00:00<00:44, 1.38it/s] Loading safetensors checkpoint shards: 3% Completed | 2/62 [00:03<02:13, 2.22s/it] ... removed ... Loading safetensors checkpoint shards: 100% Completed | 62/62 [02:58<00:00, 2.89s/it] [2025-11-07 18:06:41 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 18:06:41 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 18:06:41 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. TP MOE layer 1, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 79e5bc000b70 debug physical_to_logical_map in arg:109386352499840 moe ptr:0x637c93857590,confirm: moe->config.physical_to_logical_map:109386352499840 TP Load from loader TP MOE layer 2, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 79b5cbf36be0 debug physical_to_logical_map in arg:109386352502912 moe ptr:0x637c8dc33d40,confirm: moe->config.physical_to_logical_map:109386352502912 TP Load from loader TP MOE layer 3, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 79b183f36790 debug physical_to_logical_map in arg:109386352505984 moe ptr:0x637c8ccfc370,confirm: moe->config.physical_to_logical_map:109386352505984 TP Load from loader TP MOE layer 4, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 79ad3bf36490 debug physical_to_logical_map in arg:109386352509056 moe ptr:0x637c926482c0,confirm: moe->config.physical_to_logical_map:109386352509056 ... removed ... TP Load from loader TP MOE layer 59, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 78bff3f36490 debug physical_to_logical_map in arg:109386352678016 moe ptr:0x637c93bad1f0,confirm: moe->config.physical_to_logical_map:109386352678016 TP Load from loader TP MOE layer 60, pool: 0x637c87d2d8f0, expert num: 384, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 0 alloc 1 from other numa for 78be0bf36820 debug physical_to_logical_map in arg:109386352681088 moe ptr:0x637c901ef580,confirm: moe->config.physical_to_logical_map:109386352681088 TP Load from loader [2025-11-07 18:10:22 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=8.08 GB, mem usage=85.99 GB. [2025-11-07 18:10:22 TP0] Using KV cache dtype: torch.bfloat16 [2025-11-07 18:10:22 TP0] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 18:10:22 TP1] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 18:10:22 TP3] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 18:10:22 TP2] max_total_tokens=131072 is larger than the profiled value 101872. Use the profiled value instead. [2025-11-07 18:10:22 TP0] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 18:10:22 TP0] Memory pool end. avail mem=1.52 GB [2025-11-07 18:10:22 TP1] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 18:10:22 TP1] Memory pool end. avail mem=1.52 GB [2025-11-07 18:10:22 TP3] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 18:10:22 TP3] Memory pool end. avail mem=1.52 GB [2025-11-07 18:10:22 TP2] KV Cache is allocated. #tokens: 101872, KV size: 6.67 GB [2025-11-07 18:10:22 TP2] Memory pool end. avail mem=1.52 GB [2025-11-07 18:10:22 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 18:10:22 TP0] Capture cuda graph bs [1] [2025-11-07 18:10:22 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 18:10:22 TP3] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB [2025-11-07 18:10:22 TP2] Capture cuda graph begin. This can take up to several minutes. avail mem=1.51 GB Capturing batches (bs=1 avail_mem=1.51 GB): 0%| | 0/1 [00:00<?, ?it/s][2025-11-07 18:10:23 TP3] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 18:10:23 TP1] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 18:10:23 TP2] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-11-07 18:10:23 TP0] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=224,N=14336,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton Capturing batches (bs=1 avail_mem=1.51 GB): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.06s/it] [2025-11-07 18:10:24 TP1] Capture cuda graph end. Time elapsed: 1.62 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 18:10:24 TP3] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 18:10:24 TP2] Capture cuda graph end. Time elapsed: 1.61 s. mem usage=0.04 GB. avail mem=1.47 GB. [2025-11-07 18:10:24 TP0] Capture cuda graph end. Time elapsed: 1.65 s. mem usage=0.05 GB. avail mem=1.46 GB. [2025-11-07 18:10:25 TP3] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:10:25 TP3] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 18:10:25 TP2] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:10:25 TP2] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 18:10:25 TP1] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:10:25 TP1] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 [2025-11-07 18:10:25 TP0] Reloaded tiktoken model from /home/el_duderino/.cache/huggingface/hub/models--moonshotai--Kimi-K2-Thinking/snapshots/f1ed9ca323ee1bb8cd5fb7860f62e16a6cba32e9/tiktoken.model [2025-11-07 18:10:25 TP0] #words: 163840 - BOS ID: 163584 - EOS ID: 163585 /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( /home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/utils/hf_transformers_utils.py:433: UserWarning: Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. warnings.warn( [2025-11-07 18:10:25 TP0] max_total_num_tokens=101872, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=1, context_len=262144, available_gpu_mem=1.46 GB [2025-11-07 18:10:25] INFO: Started server process [373587] [2025-11-07 18:10:25] INFO: Waiting for application startup. [2025-11-07 18:10:25] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-11-07 18:10:25] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-11-07 18:10:25] INFO: Application startup complete. [2025-11-07 18:10:25] INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) [2025-11-07 18:10:26] INFO: 127.0.0.1:43600 - "GET /get_model_info HTTP/1.1" 200 OK [2025-11-07 18:10:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 18:10:27] INFO: 127.0.0.1:43614 - "POST /generate HTTP/1.1" 200 OK [2025-11-07 18:10:27] The server is fired up and ready to roll! [2025-11-07 18:16:37] INFO: 127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-11-07 18:16:37 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 18:16:56 TP0] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.04, #running-req: 0, #queue-req: 0, [2025-11-07 18:16:56 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 18:16:56 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 18:16:56 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 18:16:56 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1490, in forward_core return self.forward_absorb_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1772, in forward_absorb_core attn_output = self.attn_mqa( ^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_backend.py", line 838, in forward_extend self.extend_attention_fwd( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 597, in extend_attention_fwd _fwd_kernel[grid]( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/runtime/jit.py", line 617, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, ^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 498, in __getattribute__ self._init_handles() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/triton/compiler/compiler.py", line 483, in _init_handles raise OutOfResources(self.metadata.shared, max_shared, "shared memory") triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 106496, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. [2025-11-07 18:16:56] SIGQUIT received. signum=None, frame=None. It usually means one child failed. [1] 373587 killed CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --host 0.0.0.0 ```
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Nov 8, 2025):

Hi, could you please try to remove --attention-backend triton argument?

<!-- gh-comment-id:3506208317 --> @Atream commented on GitHub (Nov 8, 2025): Hi, could you please try to remove `--attention-backend triton` argument?
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@0xhaggis commented on GitHub (Nov 8, 2025):

Hi, could you please try to remove --attention-backend triton argument?

[2025-11-07 20:45:36] WARNING server_args.py:1177: Attention backend not explicitly specified. Use triton backend by default.

It did the same thing because it fell back to using triton anyway.

<!-- gh-comment-id:3506353940 --> @0xhaggis commented on GitHub (Nov 8, 2025): > Hi, could you please try to remove `--attention-backend triton` argument? `[2025-11-07 20:45:36] WARNING server_args.py:1177: Attention backend not explicitly specified. Use triton backend by default.` It did the same thing because it fell back to using triton anyway.
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Nov 8, 2025):

Maybe you can try other backend such as flashinfer. Here is the backends work for MLA.

We will also fix the Triton kernel bug in parallel.

<!-- gh-comment-id:3506370323 --> @Atream commented on GitHub (Nov 8, 2025): Maybe you can try other backend such as flashinfer. [Here](https://docs.sglang.ai/advanced_features/attention_backend.html#mla-backends) is the backends work for MLA. We will also fix the Triton kernel bug in parallel.
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@0xhaggis commented on GitHub (Nov 8, 2025):

I'd started working my way through them. torch_native failed with:

[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].
[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].
[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].
[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].
[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].
[2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

[2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core
    return self.forward_normal_core(*inner_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core
    attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend
    self._run_sdpa_forward_extend(
  File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend
    scaled_dot_product_attention(
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576].

Trying flashinfer now.

<!-- gh-comment-id:3506382708 --> @0xhaggis commented on GitHub (Nov 8, 2025): I'd started working my way through them. `torch_native` failed with: ``` [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:26 TP0] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-11-07 21:11:27 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. [2025-11-07 21:11:27 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2826, in run_scheduler_process scheduler.event_loop_overlap() File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 990, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1987, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation logits_output, can_run_cuda_graph = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2147, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 3101, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2958, in forward hidden_states, residual = layer( ^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 2684, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1397, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1484, in forward_core return self.forward_normal_core(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/models/deepseek_v2.py", line 1558, in forward_normal_core attn_output = self.attn_mha(q, k, v, forward_batch, save_kv_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 119, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 101, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 213, in forward_extend self._run_sdpa_forward_extend( File "/home/el_duderino/ktransformers/ktransformers/.venv/lib/python3.11/site-packages/sglang/srt/layers/attention/torch_native_backend.py", line 97, in _run_sdpa_forward_extend scaled_dot_product_attention( RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 192] but got: [16, 576]. ``` Trying `flashinfer` now.
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@0xhaggis commented on GitHub (Nov 8, 2025):

Flashinfer appears to have done the trick. And it's faster. Win-win. Thank you.

<!-- gh-comment-id:3506509644 --> @0xhaggis commented on GitHub (Nov 8, 2025): Flashinfer appears to have done the trick. And it's faster. Win-win. Thank you.
bsakaguchi commented 2026-10-29 14:47:44 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Nov 8, 2025):

Thank you for pointing out the issue and helping with the debugging. We’ll recommend the FlashInfer backend in the tutorial, and could you please update the post on Reddit as well?

<!-- gh-comment-id:3506522727 --> @Atream commented on GitHub (Nov 8, 2025): Thank you for pointing out the issue and helping with the debugging. We’ll recommend the FlashInfer backend in the tutorial, and could you please update the post on Reddit as well?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1038
No description provided.