kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1073] [Feature] Use Physical Cores Prior to Hyper-threads and Clearify Best Practice for Arg cpu_infer #754

New issue
Open
opened 2026-10-29 14:44:01 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 14:44:01 +00:00
Owner
Copy link

Originally created by @MotorBottle on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1073

Checklist

  • 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.

Motivation

When trying the deployment, I found how to use the cpu_infer arg needs to be clearified.

My own test was conducted on 2*E5-2686v4, 512GB RAM, RTX 3080 20g, Ubuntu22.04 using docker, same setup following official Dockerfile.

My test showed that without USE_NUMA=1, a number higher than physical cores would make inference faster (cpu_infer=68, total 36c72t). However, when USE_NUMA=1 set, cpu_infer=36 has a better performance than cpu_infer=68, but just slightly, and most of the cores in node 0 were working but many at node 1 were left idle. The futher discovery, when I turned hyperthreading off, showed the inference was greatly slowed down compared to on. And some tokens would need longer generation time than others, like 128ms compared to 98ms, one out of every 6-7 tokens, which did not happen when hyperthreading on (all were around 100ms).

Setup Performance
USE_NUMA=1, HT on, cpu_infer=56 9.65 token/s
USE_NUMA=1, HT on, cpu_infer=42 9.54 token/s
USE_NUMA=1, HT on, cpu_infer=36 9.45 token/s
USE_NUMA=1, HT on, cpu_infer=68 9.25 token/s
USE_NUMA not set, HT on, cpu_infer=68 8.2 token/s
USE_NUMA not set, HT on, cpu_infer=36 7.3 token/s
USE_NUMA not set, HT off, cpu_infer=34 6.8 token/s
USE_NUMA=1, HT off, cpu_infer=34 6.5 token/s (Needs Validation)

So I would presume, to optimize the performance, KTransformers need to have all physical cpu core threads primaryly used, then the HT threads (and of course HT on). Because when HT off, even if you set cpu_infer=cores-2, the performance would be capped somewhere. And that is for dual socket platforms with 2 NUMA nodes.

Related resources

No response

Originally created by @MotorBottle on GitHub (Apr 7, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1073 ### Checklist - [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed. ### Motivation When trying the deployment, I found how to use the `cpu_infer` arg needs to be clearified. My own test was conducted on 2*E5-2686v4, 512GB RAM, RTX 3080 20g, Ubuntu22.04 using docker, same setup following official Dockerfile. My test showed that without USE_NUMA=1, a number higher than physical cores would make inference faster (cpu_infer=68, total 36c72t). However, when `USE_NUMA=1` set, `cpu_infer=36` has a better performance than `cpu_infer=68`, but just slightly, and **most of the cores in node 0 were working but many at node 1 were left idle**. The futher discovery, when I turned hyperthreading off, showed the inference was greatly slowed down compared to on. And some tokens would need longer generation time than others, like 128ms compared to 98ms, one out of every 6-7 tokens, which did not happen when hyperthreading on (all were around 100ms). | Setup | Performance | |----------------------------------------|-------------| | USE_NUMA=1, HT on, cpu_infer=56 | 9.65 token/s | | USE_NUMA=1, HT on, cpu_infer=42 | 9.54 token/s | | USE_NUMA=1, HT on, cpu_infer=36 | 9.45 token/s | | USE_NUMA=1, HT on, cpu_infer=68 | 9.25 token/s | | USE_NUMA not set, HT on, cpu_infer=68 | 8.2 token/s | | USE_NUMA not set, HT on, cpu_infer=36 | 7.3 token/s | | USE_NUMA not set, HT off, cpu_infer=34 | 6.8 token/s | | USE_NUMA=1, HT off, cpu_infer=34 | 6.5 token/s (Needs Validation) | So I would presume, to optimize the performance, KTransformers need to have all physical cpu core threads primaryly used, then the HT threads (and of course HT on). Because when HT off, even if you set cpu_infer=cores-2, the performance would be capped somewhere. And that is for dual socket platforms with 2 NUMA nodes. ### Related resources _No response_
bsakaguchi commented 2026-10-29 14:44:02 +00:00
Author
Owner
Copy link

@yeungtuzi commented on GitHub (Apr 7, 2025):

It's very interesting and I will try it later.

<!-- gh-comment-id:2784047307 --> @yeungtuzi commented on GitHub (Apr 7, 2025): It's very interesting and I will try it later.
bsakaguchi commented 2026-10-29 14:44:02 +00:00
Author
Owner
Copy link

@magikRUKKOLA commented on GitHub (Apr 8, 2025):

Are you sure you specificed the hardware correctly? E5-2686v4 ? Its only has 18 cores. May be you have two CPUs?

Also, what model are you talking about? With what quantization? Can you provide the full command how you run it?

<!-- gh-comment-id:2785247393 --> @magikRUKKOLA commented on GitHub (Apr 8, 2025): Are you sure you specificed the hardware correctly? E5-2686v4 ? Its only has 18 cores. May be you have two CPUs? Also, what model are you talking about? With what quantization? Can you provide the full command how you run it?
bsakaguchi commented 2026-10-29 14:44:02 +00:00
Author
Owner
Copy link

@magikRUKKOLA commented on GitHub (Apr 8, 2025):

What RAM are you using?

May be you could just drop:

lshw

and

dmidecode -t memory

or something?

<!-- gh-comment-id:2785262432 --> @magikRUKKOLA commented on GitHub (Apr 8, 2025): What RAM are you using? May be you could just drop: ```bash lshw ``` and ```bash dmidecode -t memory ``` or something?
bsakaguchi commented 2026-10-29 14:44:02 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 8, 2025):

Are you sure you specificed the hardware correctly? E5-2686v4 ? Its only has 18 cores. May be you have two CPUs?

Also, what model are you talking about? With what quantization? Can you provide the full command how you run it?

I mentioned that was dual socket platform above, so in total 36 cores and 72 threads.

The model was unsloth/DeepSeek-V3-0324-UD-Q2_K_XL. Around 2.5 bit dynamically quantized.

My command:

TORCH_CUDA_ARCH_LIST="8.6" USE_NUMA=1 python3 ktransformers/server/main.py \
    --gguf_path /modelv3/ \
    --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V3-0324/snapshots/e9b33add76883f293d6bf61f6bd89b497e80e335 \
    --model_name unsloth/DeepSeek-V3-0324-UD-Q2_K_XL \
    --cpu_infer 56 \
    --max_new_tokens 4096 \
    --cache_lens 16384 \
    --temperature 0.3 \
    --cache_8bit true \
    --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
    --chunk_size 256 \
    --max_batch_size 4 \
    --backend_type balance_serve \
    --host 0.0.0.0 \
    --port 8080
<!-- gh-comment-id:2785308972 --> @MotorBottle commented on GitHub (Apr 8, 2025): > Are you sure you specificed the hardware correctly? E5-2686v4 ? Its only has 18 cores. May be you have two CPUs? > > Also, what model are you talking about? With what quantization? Can you provide the full command how you run it? I mentioned that was dual socket platform above, so in total 36 cores and 72 threads. The model was unsloth/DeepSeek-V3-0324-UD-Q2_K_XL. Around 2.5 bit dynamically quantized. My command: ``` TORCH_CUDA_ARCH_LIST="8.6" USE_NUMA=1 python3 ktransformers/server/main.py \ --gguf_path /modelv3/ \ --model_path /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V3-0324/snapshots/e9b33add76883f293d6bf61f6bd89b497e80e335 \ --model_name unsloth/DeepSeek-V3-0324-UD-Q2_K_XL \ --cpu_infer 56 \ --max_new_tokens 4096 \ --cache_lens 16384 \ --temperature 0.3 \ --cache_8bit true \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --chunk_size 256 \ --max_batch_size 4 \ --backend_type balance_serve \ --host 0.0.0.0 \ --port 8080 ```
bsakaguchi commented 2026-10-29 14:44:03 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 8, 2025):

What RAM are you using?

May be you could just drop:

lshw
and

dmidecode -t memory
or something?

My RAM configuration is 16x DDR4 ECC 2RX4 32GB 2133Mhz, 256GB for each NUMA node

<!-- gh-comment-id:2785311981 --> @MotorBottle commented on GitHub (Apr 8, 2025): > What RAM are you using? > > May be you could just drop: > > lshw > and > > dmidecode -t memory > or something? My RAM configuration is 16x DDR4 ECC 2RX4 32GB 2133Mhz, 256GB for each NUMA node
bsakaguchi commented 2026-10-29 14:44:03 +00:00
Author
Owner
Copy link

@magikRUKKOLA commented on GitHub (Apr 8, 2025):

I mentioned that was dual socket platform above, so in total 36 cores and 72 threads.

Uh. Eh. I am sorry. I misread you "Dual" as "Xeon" for some raason. :)

Can you provide the dmidecode -t memory etc. ?

<!-- gh-comment-id:2785318448 --> @magikRUKKOLA commented on GitHub (Apr 8, 2025): > I mentioned that was dual socket platform above, so in total 36 cores and 72 threads. > Uh. Eh. I am sorry. I misread you "Dual" as "Xeon" for some raason. :) Can you provide the dmidecode -t memory etc. ?
bsakaguchi commented 2026-10-29 14:44:03 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 8, 2025):

I mentioned that was dual socket platform above, so in total 36 cores and 72 threads.

Uh. Eh. I am sorry. I misread you "Dual" as "Xeon" for some raason. :)

Can you provide the dmidecode -t memory etc. ?

No problem.

# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x002B, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 256 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x002C, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x002B
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMA1
	Bank Locator: P0_Node0_Channel0_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2B5BE054
	Asset Tag: P1-DIMMA1_AssetTag (date:18/01)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x002D, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x002B
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMA2
	Bank Locator: P0_Node0_Channel0_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F387
	Asset Tag: P1-DIMMA2_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x002E, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x002B
		Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMB1
	Bank Locator: P0_Node0_Channel1_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F27C
	Asset Tag: P1-DIMMB1_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x002F, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x002B
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMB2
	Bank Locator: P0_Node0_Channel1_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F3C7
	Asset Tag: P1-DIMMB2_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0030, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 256 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x0031, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMC1
	Bank Locator: P0_Node0_Channel2_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DA48
	Asset Tag: P1-DIMMC1_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0032, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMC2
	Bank Locator: P0_Node0_Channel2_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DB31
	Asset Tag: P1-DIMMC2_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0033, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMD1
	Bank Locator: P0_Node0_Channel3_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DA71
	Asset Tag: P1-DIMMD1_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0034, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0030
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P1-DIMMD2
	Bank Locator: P0_Node0_Channel3_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DBE4
	Asset Tag: P1-DIMMD2_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0035, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 256 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x0036, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0035
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMME1
	Bank Locator: P1_Node1_Channel0_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F3E4
	Asset Tag: P2-DIMME1_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0037, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0035
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMME2
	Bank Locator: P1_Node1_Channel0_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F23E
	Asset Tag: P2-DIMME2_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0038, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0035
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMF1
	Bank Locator: P1_Node1_Channel1_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F383
	Asset Tag: P2-DIMMF1_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x0039, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0035
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMF2
	Bank Locator: P1_Node1_Channel1_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 2AE6F3CA
	Asset Tag: P2-DIMMF2_AssetTag (date:17/44)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x003A, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 256 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x003B, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x003A
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMG1
	Bank Locator: P1_Node1_Channel2_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162D9DC
	Asset Tag: P2-DIMMG1_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x003C, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x003A
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMG2
	Bank Locator: P1_Node1_Channel2_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DA6A
	Asset Tag: P2-DIMMG2_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x003D, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x003A
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMH1
	Bank Locator: P1_Node1_Channel3_Dimm0
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DB4A
	Asset Tag: P2-DIMMH1_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown

Handle 0x003E, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x003A
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: None
	Locator: P2-DIMMH2
	Bank Locator: P1_Node1_Channel3_Dimm1
	Type: DDR4
	Type Detail: Synchronous
	Speed: 2400 MT/s
	Manufacturer: SK Hynix
	Serial Number: 1162DAFE
	Asset Tag: P2-DIMMH2_AssetTag (date:17/07)
	Part Number: HMA84GR7AFR4N-UH   
	Rank: 2
	Configured Memory Speed: 2133 MT/s
	Minimum Voltage: Unknown
	Maximum Voltage: Unknown
	Configured Voltage: Unknown
<!-- gh-comment-id:2785320673 --> @MotorBottle commented on GitHub (Apr 8, 2025): > > I mentioned that was dual socket platform above, so in total 36 cores and 72 threads. > > Uh. Eh. I am sorry. I misread you "Dual" as "Xeon" for some raason. :) > > Can you provide the dmidecode -t memory etc. ? No problem. ``` # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.0.0 present. Handle 0x002B, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 256 GB Error Information Handle: Not Provided Number Of Devices: 4 Handle 0x002C, DMI type 17, 40 bytes Memory Device Array Handle: 0x002B Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMA1 Bank Locator: P0_Node0_Channel0_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2B5BE054 Asset Tag: P1-DIMMA1_AssetTag (date:18/01) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x002D, DMI type 17, 40 bytes Memory Device Array Handle: 0x002B Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMA2 Bank Locator: P0_Node0_Channel0_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F387 Asset Tag: P1-DIMMA2_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x002E, DMI type 17, 40 bytes Memory Device Array Handle: 0x002B Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMB1 Bank Locator: P0_Node0_Channel1_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F27C Asset Tag: P1-DIMMB1_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x002F, DMI type 17, 40 bytes Memory Device Array Handle: 0x002B Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMB2 Bank Locator: P0_Node0_Channel1_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F3C7 Asset Tag: P1-DIMMB2_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0030, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 256 GB Error Information Handle: Not Provided Number Of Devices: 4 Handle 0x0031, DMI type 17, 40 bytes Memory Device Array Handle: 0x0030 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMC1 Bank Locator: P0_Node0_Channel2_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DA48 Asset Tag: P1-DIMMC1_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0032, DMI type 17, 40 bytes Memory Device Array Handle: 0x0030 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMC2 Bank Locator: P0_Node0_Channel2_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DB31 Asset Tag: P1-DIMMC2_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0033, DMI type 17, 40 bytes Memory Device Array Handle: 0x0030 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMD1 Bank Locator: P0_Node0_Channel3_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DA71 Asset Tag: P1-DIMMD1_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0034, DMI type 17, 40 bytes Memory Device Array Handle: 0x0030 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P1-DIMMD2 Bank Locator: P0_Node0_Channel3_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DBE4 Asset Tag: P1-DIMMD2_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0035, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 256 GB Error Information Handle: Not Provided Number Of Devices: 4 Handle 0x0036, DMI type 17, 40 bytes Memory Device Array Handle: 0x0035 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMME1 Bank Locator: P1_Node1_Channel0_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F3E4 Asset Tag: P2-DIMME1_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0037, DMI type 17, 40 bytes Memory Device Array Handle: 0x0035 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMME2 Bank Locator: P1_Node1_Channel0_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F23E Asset Tag: P2-DIMME2_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0038, DMI type 17, 40 bytes Memory Device Array Handle: 0x0035 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMF1 Bank Locator: P1_Node1_Channel1_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F383 Asset Tag: P2-DIMMF1_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x0039, DMI type 17, 40 bytes Memory Device Array Handle: 0x0035 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMF2 Bank Locator: P1_Node1_Channel1_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 2AE6F3CA Asset Tag: P2-DIMMF2_AssetTag (date:17/44) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x003A, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 256 GB Error Information Handle: Not Provided Number Of Devices: 4 Handle 0x003B, DMI type 17, 40 bytes Memory Device Array Handle: 0x003A Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMG1 Bank Locator: P1_Node1_Channel2_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162D9DC Asset Tag: P2-DIMMG1_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x003C, DMI type 17, 40 bytes Memory Device Array Handle: 0x003A Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMG2 Bank Locator: P1_Node1_Channel2_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DA6A Asset Tag: P2-DIMMG2_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x003D, DMI type 17, 40 bytes Memory Device Array Handle: 0x003A Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMH1 Bank Locator: P1_Node1_Channel3_Dimm0 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DB4A Asset Tag: P2-DIMMH1_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x003E, DMI type 17, 40 bytes Memory Device Array Handle: 0x003A Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: P2-DIMMH2 Bank Locator: P1_Node1_Channel3_Dimm1 Type: DDR4 Type Detail: Synchronous Speed: 2400 MT/s Manufacturer: SK Hynix Serial Number: 1162DAFE Asset Tag: P2-DIMMH2_AssetTag (date:17/07) Part Number: HMA84GR7AFR4N-UH Rank: 2 Configured Memory Speed: 2133 MT/s Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown ```
bsakaguchi commented 2026-10-29 14:44:03 +00:00
Author
Owner
Copy link

@magikRUKKOLA commented on GitHub (Apr 8, 2025):

Configured Memory Speed: 2133 MT/s

Very interesting! But why its configured lower than it can run (2400 MT/s)?

<!-- gh-comment-id:2785556698 --> @magikRUKKOLA commented on GitHub (Apr 8, 2025): > Configured Memory Speed: 2133 MT/s Very interesting! But why its configured lower than it can run (2400 MT/s)?
bsakaguchi commented 2026-10-29 14:44:04 +00:00
Author
Owner
Copy link

@MotorBottle commented on GitHub (Apr 8, 2025):

Configured Memory Speed: 2133 MT/s

Very interesting! But why its configured lower than it can run (2400 MT/s)?

Motherboard limits

<!-- gh-comment-id:2785560144 --> @MotorBottle commented on GitHub (Apr 8, 2025): > > Configured Memory Speed: 2133 MT/s > > Very interesting! But why its configured lower than it can run (2400 MT/s)? Motherboard limits
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#754
No description provided.