[GH-ISSUE #138 ] 想在家庭实验室中配备 请教几个配置问题 #77

Author

Owner

@edward-ng-duke commented on GitHub (Feb 11, 2025):

cpu性能会有较大影响，核心数量和计算带宽都会影响cpu端的速度。如果想试用我们的双socket加速，需要双路。显卡可以不用很好，但是至少需要16g显存，和ampere以上架构（30系列以上）。

Dual socket will need at least 1T DRAM ( Or it will use swap area with SSD which will cause a great performance drop)双插槽至少需要 1T DRAM（否则将使用与 SSD 的交换区，这将导致性能大幅下降）

@KMSorSMS Is the reason that 1T DDR5 RAM is needed for the dual slots in the case of Q4_K_M because of the 2 NUMA nodes? If I want to run the Q8 version, should I get 2T of RAM ready?
分享

@edward-ng-duke commented on GitHub (Feb 11, 2025): > > cpu性能会有较大影响，核心数量和计算带宽都会影响cpu端的速度。如果想试用我们的双socket加速，需要双路。显卡可以不用很好，但是至少需要16g显存，和ampere以上架构（30系列以上）。 > > Dual socket will need at least 1T DRAM ( Or it will use swap area with SSD which will cause a great performance drop)双插槽至少需要 1T DRAM（否则将使用与 SSD 的交换区，这将导致性能大幅下降） @KMSorSMS Is the reason that 1T DDR5 RAM is needed for the dual slots in the case of Q4_K_M because of the 2 NUMA nodes? If I want to run the Q8 version, should I get 2T of RAM ready? 分享

bsakaguchi commented

Author

Owner

@edward-ng-duke commented on GitHub (Feb 11, 2025):

I'm sorry, I'd like to add something. I've noticed that Llama.cpp has very high requirements for the number of memory channels and the memory frequency. For example, it requires 24 channels and DDR5 5600MHz. I'd like to ask whether ktransformers has any dependencies on these two aspects.

@edward-ng-duke commented on GitHub (Feb 11, 2025): I'm sorry, I'd like to add something. I've noticed that Llama.cpp has very high requirements for the number of memory channels and the memory frequency. For example, it requires 24 channels and DDR5 5600MHz. I'd like to ask whether ktransformers has any dependencies on these two aspects.

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 11, 2025):

cpu性能会有较大影响，核心数量和计算带宽都会影响cpu端的速度。如果想试用我们的双socket加速，需要双路。显卡可以不用很好，但是至少需要16g显存，和ampere以上架构（30系列以上）。

Dual socket will need at least 1T DRAM ( Or it will use swap area with SSD which will cause a great performance drop)双插槽至少需要 1T DRAM（否则将使用与 SSD 的交换区，这将导致性能大幅下降）

@KMSorSMS Is the reason that 1T DDR5 RAM is needed for the dual slots in the case of Q4_K_M because of the 2 NUMA nodes? If I want to run the Q8 version, should I get 2T of RAM ready?
I'm sorry, I'd like to add something. I've noticed that Llama.cpp has very high requirements for the number of memory channels and the memory frequency. For example, it requires 24 channels and DDR5 5600MHz. I'd like to ask whether ktransformers has any dependencies on these two aspects.

Yes. This is because we currently avoid the communication overhead between nodes by using redundant weights for two numa nodes.

And We don't have dependencies on channels and memory frequency. But the better the faster, and we use 8 channels and standard DDR5-4800 server DRAM

@KMSorSMS commented on GitHub (Feb 11, 2025): > > > cpu性能会有较大影响，核心数量和计算带宽都会影响cpu端的速度。如果想试用我们的双socket加速，需要双路。显卡可以不用很好，但是至少需要16g显存，和ampere以上架构（30系列以上）。 > > > > > > Dual socket will need at least 1T DRAM ( Or it will use swap area with SSD which will cause a great performance drop)双插槽至少需要 1T DRAM（否则将使用与 SSD 的交换区，这将导致性能大幅下降） > > [@KMSorSMS](https://github.com/KMSorSMS) Is the reason that 1T DDR5 RAM is needed for the dual slots in the case of Q4_K_M because of the 2 NUMA nodes? If I want to run the Q8 version, should I get 2T of RAM ready? > I'm sorry, I'd like to add something. I've noticed that Llama.cpp has very high requirements for the number of memory channels and the memory frequency. For example, it requires 24 channels and DDR5 5600MHz. I'd like to ask whether ktransformers has any dependencies on these two aspects. Yes. This is because we currently avoid the communication overhead between nodes by using redundant weights for two numa nodes.<br> And We don't have dependencies on channels and memory frequency. But the better the faster, and we use 8 channels and standard DDR5-4800 server DRAM

bsakaguchi commented

Author

Owner

@optimustian commented on GitHub (Feb 11, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch

我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

@optimustian commented on GitHub (Feb 11, 2025): 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

bsakaguchi commented

Author

Owner

@buliaoyin commented on GitHub (Feb 11, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch

我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

@buliaoyin commented on GitHub (Feb 11, 2025): > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

bsakaguchi commented

Author

Owner

@bltcn commented on GitHub (Feb 12, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

怎么修改，能否给个教程学习一下，谢谢

@bltcn commented on GitHub (Feb 12, 2025): > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍怎么修改，能否给个教程学习一下，谢谢

bsakaguchi commented

Author

Owner

@wxtt522 commented on GitHub (Feb 13, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

请问一下，q2精度的推理效果如何，和q4差别大吗？我看b站评测1.58b的效果甚至不如70b的蒸馏版

@wxtt522 commented on GitHub (Feb 13, 2025): > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍请问一下，q2精度的推理效果如何，和q4差别大吗？我看b站评测1.58b的效果甚至不如70b的蒸馏版

bsakaguchi commented

Author

Owner

@buliaoyin commented on GitHub (Feb 13, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

请问一下，q2精度的推理效果如何，和q4差别大吗？我看b站评测1.58b的效果甚至不如70b的蒸馏版

在我的使用场景下感受不太到特别大的差异，我是跟官方API比的

@buliaoyin commented on GitHub (Feb 13, 2025): > > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > > > > > 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍 > > 请问一下，q2精度的推理效果如何，和q4差别大吗？我看b站评测1.58b的效果甚至不如70b的蒸馏版在我的使用场景下感受不太到特别大的差异，我是跟官方API比的

bsakaguchi commented

Author

Owner

@Refarin commented on GitHub (Feb 13, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

能否问下是什么主板、机箱和内存是什么？我想搭一个

@Refarin commented on GitHub (Feb 13, 2025): > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍能否问下是什么主板、机箱和内存是什么？我想搭一个

bsakaguchi commented

Author

Owner

@dragen1860 commented on GitHub (Feb 14, 2025):

@buliaoyin 想问下怎么改掉marlin呢，谢谢

@dragen1860 commented on GitHub (Feb 14, 2025): @buliaoyin 想问下怎么改掉marlin呢，谢谢

bsakaguchi commented

Author

Owner

@vc5409ftu commented on GitHub (Feb 15, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍

能否问下是什么主板、机箱和内存是什么？我想搭一个

可以参考
t5810/5820准系统，256g2666内存

@vc5409ftu commented on GitHub (Feb 15, 2025): > > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > > > > > 感谢说明，改掉Marlin后可以跑了，双路2680V4 + 64Gx8 D4 2400+2 2080Ti，跑UD-Q2_K_XL有5.2Tok/s，llamacpp是2.6Tok/s，刚好一倍 > > 能否问下是什么主板、机箱和内存是什么？我想搭一个可以参考 t5810/5820准系统，256g2666内存

bsakaguchi commented

Author

Owner

@shaofanqi commented on GitHub (Feb 21, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch

我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀

@shaofanqi commented on GitHub (Feb 21, 2025): > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s 您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀

bsakaguchi commented

Author

Owner

@ChenYFan commented on GitHub (Feb 25, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀

Qwen2MoE貌似强制启用了FlashAttn2

@ChenYFan commented on GitHub (Feb 25, 2025): > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > 您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀 ![Image](https://github.com/user-attachments/assets/aaa7214f-8e55-4e23-84a6-c715abfcb7e3) Qwen2MoE貌似强制启用了FlashAttn2

bsakaguchi commented

Author

Owner

@363843342 commented on GitHub (Feb 25, 2025):

KLinearMarlin改为KLinearTorch支持1080ti p100的十系老卡吗？

@363843342 commented on GitHub (Feb 25, 2025): KLinearMarlin改为KLinearTorch支持1080ti p100的十系老卡吗？

bsakaguchi commented

Author

Owner

@363843342 commented on GitHub (Feb 25, 2025):

2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch
我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s

您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀

flash_attn 2不再支持20系，可能安装flash_attn 1.0.9老版本可以解决？

@363843342 commented on GitHub (Feb 25, 2025): > > 2080ti需要把optimize_rule里的KLinearMarlin改为KLinearTorch > > 我双路EPYC 7601 + 512G DDR4 2133 + 两张2080ti 用这个库跑Q4速度2tok/s，llama.cpp是0.6 tok/s，提升三倍多。llama.cpp跑1.58bit是1.2tok/s > > 您好！我的2080ti出现了RuntimeError: FlashAttention only supports Ampere GPUs or newer.，在运行中使用了--no_flash_atten true 也不行呀 flash_attn 2不再支持20系，可能安装flash_attn 1.0.9老版本可以解决？

bsakaguchi commented