mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1050] [Feature] capturing cuda graph加载比较慢 #740
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#740
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @344303947 on GitHub (Apr 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1050
Checklist
Motivation
到这一步的时候加载比较慢，有办法优化吗
loading output_norm.weight to cuda
capturing cuda graph 1 1
2025-04-05 12:17:54,410 - INFO - flashinfer.jit: Loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-04-05 12:17:54,431 - INFO - flashinfer.jit: Finished loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False
2025-04-05 12:17:54,457 - INFO - flashinfer.jit: Loading JIT ops: norm
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-04-05 12:17:54,474 - INFO - flashinfer.jit: Finished loading JIT ops: norm
cuda_graph: 1/6, warmup finished.
capturing cuda graph 2 2
cuda_graph: 2/6, warmup finished.
capturing cuda graph 3 3
cuda_graph: 3/6, warmup finished.
capturing cuda graph 4 4
cuda_graph: 4/6, warmup finished.
capturing cuda graph 4 64
cuda_graph: 5/6, warmup finished.
capturing cuda graph 4 256
cuda_graph: 6/6, warmup finished.
Related resources
No response
@qiyuxinlin commented on GitHub (Apr 5, 2025):
您好，我们开了 [1,2,3,4,64,256] 这几组 cuda graph，其中 256 对应 chunk_size，您可以调整 chunk_size 来加速 capture 流程，但是这可能会影响 prefill 速度
@344303947 commented on GitHub (Apr 5, 2025):
好的感谢告知，就是对比以前的版本 load 完成就可以对话了，感觉时间长了很多。
@344303947 commented on GitHub (Apr 5, 2025):
--max_new_tokens 2024
--cache_lens 32768
设置成 --max_new_tokens 32000 \ 如果设置过高，就不会并发执行了，不知道是什么问题
@yeungtuzi commented on GitHub (Apr 10, 2025):
需要cache_lens >= 并发数*max_new_tokens