[PR #852] [MERGED] cpuinfer: filter repeated backend instantiation #1501

New issue

Closed

opened 2026-10-29 14:52:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:52:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/852
Author: @Lander-Hatsune
Created: 3/10/2025
Status: ✅ Merged
Merged: 3/14/2025
Merged by: @Atream

Base: main ← Head: main

📝 Commits (1)

d166fb9 cpuinfer: filter repeated backend instantiation

📊 Changes

1 file changed (+6 additions, -1 deletions)

View changed files

📝 ktransformers/operators/cpuinfer.py (+6 -1)

📄 Description

Currently CPUInfer will be instantiated twice as class(or static) attributes of the two CPU operator (KLinearCPUInfer and KExpertsCPU), if both files are imported.

# ktransformers/operators/linear.py:362
class KLinearCPUInfer(KLinearBase):
    CPU_INFER = CPUInfer(Config().cpu_infer)

# ktransformers/operators/experts.py:115
class KExpertsCPU(KExpertsBase):
    input_tensor_cpu:Tensor = None
    expert_ids_cpu:Tensor = None
    weights_cpu:Tensor = None
    output_cpu:Tensor = None
    output_gpu_map:dict = {} # Manage output tensor buffer on different gpu
    #stream_map:dict = {} # Manage cuda stream on different gpu
    #gguf_loader:GGUFLoader = None
    CPU_INFER = CPUInfer(Config().cpu_infer)

Considering __init__ method of class CPUInfer, this repetition leads to multiple instantiation of Backends (also the worker threads) and TaskQueues.

This commit filters repeated CPUInfer.__init__()s by maintaining a maximum number of threads requested.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/852 **Author:** [@Lander-Hatsune](https://github.com/Lander-Hatsune) **Created:** 3/10/2025 **Status:** ✅ Merged **Merged:** 3/14/2025 **Merged by:** [@Atream](https://github.com/Atream) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`d166fb9`](https://github.com/kvcache-ai/ktransformers/commit/d166fb9f6e94483bf0cf222eaeb95fd1a04df9d8) cpuinfer: filter repeated backend instantiation ### 📊 Changes **1 file changed** (+6 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `ktransformers/operators/cpuinfer.py` (+6 -1) </details> ### 📄 Description Currently `CPUInfer` will be instantiated twice as **class(or static) attributes** of the two CPU operator (`KLinearCPUInfer` and `KExpertsCPU`), if both files are imported. ```py # ktransformers/operators/linear.py:362 class KLinearCPUInfer(KLinearBase): CPU_INFER = CPUInfer(Config().cpu_infer) # ktransformers/operators/experts.py:115 class KExpertsCPU(KExpertsBase): input_tensor_cpu:Tensor = None expert_ids_cpu:Tensor = None weights_cpu:Tensor = None output_cpu:Tensor = None output_gpu_map:dict = {} # Manage output tensor buffer on different gpu #stream_map:dict = {} # Manage cuda stream on different gpu #gguf_loader:GGUFLoader = None CPU_INFER = CPUInfer(Config().cpu_infer) ``` Considering `__init__` method of class `CPUInfer`, this repetition leads to multiple instantiation of `Backend`s (also the worker threads) and `TaskQueue`s. This commit filters repeated `CPUInfer.__init__()`s by maintaining a maximum number of threads requested. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:52:58 +00:00