kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #888] [Bug] 推理时报错：-sizes[i] <= index && index < sizes[i] && "index out of bounds" #641

New issue
Closed
opened 2026-10-29 14:42:17 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-29 14:42:17 +00:00
Owner
Copy link

Originally created by @minmie on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/888

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

启动命令：

nohup ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/models/unsloth/DeepSeek-R1-GGUF --port 10002 --cpu_infer 65 --max_new_tokens 4000 --force_think >> kt.log 2>&1 &

在推理过程中，总是会突然报错然后服务直接挂掉

2025-03-13 16:30:00,607 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 7324])
<think>
2025-03-13 16:30:00,608 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[133]: input_ids: torch.Size([1, 7326])
2025-03-13 16:30:00,608 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[159]: same prefix len: 3
2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[168]: input_ids: torch.Size([1, 7323])
2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[170]: generate_ids: torch.Size([1, 3])
2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[181]: cache position: 3 to 7326
嗯，用户让我分析这段聊天记录，找出可能的舞弊行为和其他相关信息。......................（手动省略的）专注于工作相关的讨论。
</think>

### 1：双方关系
**关系判断**：同事关系（概率95%）
**依据**：
1. 存在工作协同关系（多次讨论客户对接、合同处理、数据报表等具体业务内容）
2. 对话中涉及共同上级（光哥、龙哥）、跨部门协作（法务、财务）
3. 使用公司内部术语（如"xxx"、"PLC"、"负向环节"）
4. 对话中频繁提及绩效考核、转正目标、工作交接等职场内容

---

### 2：用户聊天话题类型TOP5
| 话题类型 | 出现次数 | 示例内容 |
|---------|----------|----------|
| 客户合同处理 | 8次 | xxx补充协议、简历点修改、法务流程（8月1日-8月23日） |
| 数据看板运营 | 6次 | 简历成本趋势分析、消耗数据拆解（7月5日-8月3日） |
| xxxx调整 | 5次 | xxx单替换、代工厂../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [1,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [2,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [3,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [4,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [5,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [6,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.

Reproduction

使用命令启动后，进行推理时，会报错。

Environment

GPU
Image

OS
Image

cpu


架构：                    x86_64
  CPU 运行模式：          32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  字节序：                Little Endian
CPU:                      128
  在线 CPU 列表：         0-127
厂商 ID：                 GenuineIntel
  型号名称：              INTEL(R) XEON(R) GOLD 6530
    CPU 系列：            6
    型号：                207
    每个核的线程数：      2
    每个座的核数：        32
    座：                  2
    步进：                2
    CPU(s) scaling MHz:   25%
    CPU 最大 MHz：        4000.0000
    CPU 最小 MHz：        800.0000
    BogoMIPS：            4200.00
    标记：                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon p
                          ebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic
                          movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tp
                          r_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd s
                          ha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp
                           hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_dete
                          ct cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization features:
  虚拟化：                VT-x
Caches (sum of all):
  L1d:                    3 MiB (64 instances)
  L1i:                    2 MiB (64 instances)
  L2:                     128 MiB (64 instances)
  L3:                     320 MiB (2 instances)
NUMA:
  NUMA 节点：             2
  NUMA 节点0 CPU：        0-31,64-95
  NUMA 节点1 CPU：        32-63,96-127
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Vulnerable: eIBRS with unprivileged eBPF
  Srbds:                  Not affected
  Tsx async abort:        Not affected

pip 包信息


Package                  Version
------------------------ --------------------------
accelerate               1.4.0
aiohappyeyeballs         2.4.6
aiohttp                  3.11.13
aiosignal                1.3.2
annotated-types          0.7.0
anyio                    4.8.0
attrs                    25.1.0
autocommand              2.2.2
backports.tarfile        1.2.0
blessed                  1.20.0
blobfile                 3.0.0
build                    1.2.2.post1
certifi                  2025.1.31
charset-normalizer       3.4.1
click                    8.1.8
colorlog                 6.9.0
cpufeature               0.2.1
einops                   0.8.1
fastapi                  0.115.8
filelock                 3.17.0
fire                     0.7.0
flash_attn               2.7.4.post1
flash_attn               2.7.4.post1
frozenlist               1.5.0
fsspec                   2025.2.0
greenlet                 3.1.1
h11                      0.14.0
httpcore                 1.0.7
httpx                    0.28.1
huggingface-hub          0.29.1
idna                     3.10
importlib_metadata       8.0.0
inflect                  7.3.1
inquirerpy               0.3.4
jaraco.collections       5.1.0
jaraco.context           5.3.0
jaraco.functools         4.0.1
jaraco.text              3.12.1
Jinja2                   3.1.5
jsonpatch                1.33
jsonpointer              3.0.0
ktransformers            0.2.2rc1+cu124torch24fancy
langchain                0.2.17
langchain-core           0.2.43
langchain-text-splitters 0.2.4
langsmith                0.1.147
lxml                     5.3.1
MarkupSafe               3.0.2
more-itertools           10.3.0
mpmath                   1.3.0
multidict                6.1.0
networkx                 3.4.2
ninja                    1.11.1.3
numpy                    1.26.4
nvidia-cublas-cu12       12.4.2.65
nvidia-cuda-cupti-cu12   12.4.99
nvidia-cuda-nvrtc-cu12   12.4.99
nvidia-cuda-runtime-cu12 12.4.99
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.2.0.44
nvidia-curand-cu12       10.3.5.119
nvidia-cusolver-cu12     11.6.0.99
nvidia-cusparse-cu12     12.3.0.142
nvidia-nccl-cu12         2.20.5
nvidia-nvjitlink-cu12    12.4.99
nvidia-nvtx-cu12         12.4.99
orjson                   3.10.15
packaging                24.2
pfzy                     0.3.4
pillow                   11.1.0
pip                      25.0
platformdirs             4.2.2
prompt_toolkit           3.0.50
propcache                0.3.0
protobuf                 5.29.3
psutil                   7.0.0
pycryptodomex            3.21.0
pydantic                 2.10.6
pydantic_core            2.27.2
pyproject_hooks          1.2.0
PyYAML                   6.0.2
regex                    2024.11.6
requests                 2.32.3
requests-toolbelt        1.0.0
safetensors              0.5.2
sentencepiece            0.2.0
setuptools               75.8.0
six                      1.17.0
sniffio                  1.3.1
SQLAlchemy               2.0.38
starlette                0.45.3
sympy                    1.13.3
tenacity                 8.5.0
termcolor                2.5.0
tiktoken                 0.9.0
tokenizers               0.19.1
tomli                    2.0.1
torch                    2.4.1+cu124
torchaudio               2.4.1+cu124
torchvision              0.19.1+cu124
tqdm                     4.67.1
transformers             4.43.2
triton                   3.0.0
typeguard                4.3.0
typing_extensions        4.12.2
urllib3                  2.3.0
uvicorn                  0.34.0
wcwidth                  0.2.13
wheel                    0.45.1
yarl                     1.18.3
zipp                     3.19.2
Originally created by @minmie on GitHub (Mar 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/888 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug 启动命令： ``` nohup ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/models/unsloth/DeepSeek-R1-GGUF --port 10002 --cpu_infer 65 --max_new_tokens 4000 --force_think >> kt.log 2>&1 & ``` 在推理过程中，总是会突然报错然后服务直接挂掉 ``` 2025-03-13 16:30:00,607 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 7324]) <think> 2025-03-13 16:30:00,608 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[133]: input_ids: torch.Size([1, 7326]) 2025-03-13 16:30:00,608 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[159]: same prefix len: 3 2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[168]: input_ids: torch.Size([1, 7323]) 2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[170]: generate_ids: torch.Size([1, 3]) 2025-03-13 16:30:00,609 DEBUG /home/chenjq/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[181]: cache position: 3 to 7326 嗯，用户让我分析这段聊天记录，找出可能的舞弊行为和其他相关信息。......................（手动省略的）专注于工作相关的讨论。 </think> ### 1：双方关系 **关系判断**：同事关系（概率95%） **依据**： 1. 存在工作协同关系（多次讨论客户对接、合同处理、数据报表等具体业务内容） 2. 对话中涉及共同上级（光哥、龙哥）、跨部门协作（法务、财务） 3. 使用公司内部术语（如"xxx"、"PLC"、"负向环节"） 4. 对话中频繁提及绩效考核、转正目标、工作交接等职场内容 --- ### 2：用户聊天话题类型TOP5 | 话题类型 | 出现次数 | 示例内容 | |---------|----------|----------| | 客户合同处理 | 8次 | xxx补充协议、简历点修改、法务流程（8月1日-8月23日） | | 数据看板运营 | 6次 | 简历成本趋势分析、消耗数据拆解（7月5日-8月3日） | | xxxx调整 | 5次 | xxx单替换、代工厂../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [1,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [2,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [3,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [4,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [5,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [6,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ``` ### Reproduction 使用命令启动后，进行推理时，会报错。 ### Environment GPU ![Image](https://github.com/user-attachments/assets/e5a5fda8-2d0e-46dd-a3c5-07e5961af055) OS ![Image](https://github.com/user-attachments/assets/18b2f981-e34a-4c89-9a8a-0ecd25e7d0c5) cpu ``` 架构： x86_64 CPU 运行模式： 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual 字节序： Little Endian CPU: 128 在线 CPU 列表： 0-127 厂商 ID： GenuineIntel 型号名称： INTEL(R) XEON(R) GOLD 6530 CPU 系列： 6 型号： 207 每个核的线程数： 2 每个座的核数： 32 座： 2 步进： 2 CPU(s) scaling MHz: 25% CPU 最大 MHz： 4000.0000 CPU 最小 MHz： 800.0000 BogoMIPS： 4200.00 标记： fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon p ebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tp r_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd s ha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_dete ct cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Virtualization features: 虚拟化： VT-x Caches (sum of all): L1d: 3 MiB (64 instances) L1i: 2 MiB (64 instances) L2: 128 MiB (64 instances) L3: 320 MiB (2 instances) NUMA: NUMA 节点： 2 NUMA 节点0 CPU： 0-31,64-95 NUMA 节点1 CPU： 32-63,96-127 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Vulnerable: eIBRS with unprivileged eBPF Srbds: Not affected Tsx async abort: Not affected ``` pip 包信息 ``` Package Version ------------------------ -------------------------- accelerate 1.4.0 aiohappyeyeballs 2.4.6 aiohttp 3.11.13 aiosignal 1.3.2 annotated-types 0.7.0 anyio 4.8.0 attrs 25.1.0 autocommand 2.2.2 backports.tarfile 1.2.0 blessed 1.20.0 blobfile 3.0.0 build 1.2.2.post1 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 colorlog 6.9.0 cpufeature 0.2.1 einops 0.8.1 fastapi 0.115.8 filelock 3.17.0 fire 0.7.0 flash_attn 2.7.4.post1 flash_attn 2.7.4.post1 frozenlist 1.5.0 fsspec 2025.2.0 greenlet 3.1.1 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.29.1 idna 3.10 importlib_metadata 8.0.0 inflect 7.3.1 inquirerpy 0.3.4 jaraco.collections 5.1.0 jaraco.context 5.3.0 jaraco.functools 4.0.1 jaraco.text 3.12.1 Jinja2 3.1.5 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.2.2rc1+cu124torch24fancy langchain 0.2.17 langchain-core 0.2.43 langchain-text-splitters 0.2.4 langsmith 0.1.147 lxml 5.3.1 MarkupSafe 3.0.2 more-itertools 10.3.0 mpmath 1.3.0 multidict 6.1.0 networkx 3.4.2 ninja 1.11.1.3 numpy 1.26.4 nvidia-cublas-cu12 12.4.2.65 nvidia-cuda-cupti-cu12 12.4.99 nvidia-cuda-nvrtc-cu12 12.4.99 nvidia-cuda-runtime-cu12 12.4.99 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.2.0.44 nvidia-curand-cu12 10.3.5.119 nvidia-cusolver-cu12 11.6.0.99 nvidia-cusparse-cu12 12.3.0.142 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.4.99 nvidia-nvtx-cu12 12.4.99 orjson 3.10.15 packaging 24.2 pfzy 0.3.4 pillow 11.1.0 pip 25.0 platformdirs 4.2.2 prompt_toolkit 3.0.50 propcache 0.3.0 protobuf 5.29.3 psutil 7.0.0 pycryptodomex 3.21.0 pydantic 2.10.6 pydantic_core 2.27.2 pyproject_hooks 1.2.0 PyYAML 6.0.2 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.2 sentencepiece 0.2.0 setuptools 75.8.0 six 1.17.0 sniffio 1.3.1 SQLAlchemy 2.0.38 starlette 0.45.3 sympy 1.13.3 tenacity 8.5.0 termcolor 2.5.0 tiktoken 0.9.0 tokenizers 0.19.1 tomli 2.0.1 torch 2.4.1+cu124 torchaudio 2.4.1+cu124 torchvision 0.19.1+cu124 tqdm 4.67.1 transformers 4.43.2 triton 3.0.0 typeguard 4.3.0 typing_extensions 4.12.2 urllib3 2.3.0 uvicorn 0.34.0 wcwidth 0.2.13 wheel 0.45.1 yarl 1.18.3 zipp 3.19.2 ```
bsakaguchi commented 2026-10-29 14:42:18 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Mar 15, 2025):

请设置cache_lens参数来设置最长的上下文大小

<!-- gh-comment-id:2726444422 --> @Atream commented on GitHub (Mar 15, 2025): 请设置cache_lens参数来设置最长的上下文大小
bsakaguchi commented 2026-10-29 14:42:18 +00:00
Author
Owner
Copy link

@minmie commented on GitHub (Mar 24, 2025):

@Atream
谢谢，按照要求，增加了cache_lens，目前看一切正常。

<!-- gh-comment-id:2746767034 --> @minmie commented on GitHub (Mar 24, 2025): @Atream 谢谢，按照要求，增加了cache_lens，目前看一切正常。
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#641
No description provided.