[GH-ISSUE #2208] A100 not support? #1302

New issue

Open

opened 2026-10-29 14:51:00 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:51:00 +00:00

Owner

Originally created by @SunYong0821 on GitHub (Sep 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2208

Reminder

I have read the above rules and searched the existing issues.

Description

why？？？A100 not support?

Pull Request

No response

Originally created by @SunYong0821 on GitHub (Sep 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2208 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description <img width="940" height="618" alt="Image" src="https://github.com/user-attachments/assets/1d8184d3-0553-4980-a1ea-693d1b8bb3dc" /> why？？？A100 not support? ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:51:00 +00:00

bsakaguchi commented

2026-10-29 14:51:01 +00:00

Author

Owner

@AGenchev commented on GitHub (Sep 19, 2026):

In general, they support A-100. For example, this works on 4xA-100, 2x24-core AVX-512 CPUs:

python -m sglang.launch_server \
  --model-path '/home/user/LLMs/weights/GLM-5.2-FP16' \
  --kt-weight-path '/home/user/LLMs/weights/GLM-5.2-FP16' \
  --kt-cpuinfer 48 \
  --kt-threadpool-count 8 \
  --sleep-on-idle \
  --kt-num-gpu-experts 15 \
  --kt-method BF16 \
  --kt-gpu-prefill-token-threshold 256 \
  --kt-enable-dynamic-expert-update \
  --kt-expert-placement-strategy uniform \
  --enable-p2p-check \
  --enable-mixed-chunk \
  --watchdog-timeout 3000 \
  --tp-size 4 \
  --cuda-graph-max-bs 2 \
  --trust-remote-code \
  --mem-fraction-static 0.95 \
  --kv-cache-dtype auto \
  --max-total-tokens 192000 \
  --max-running-requests 2 \
  --attention-backend triton \
  --fp8-gemm-backend auto \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --served-model-name IO_model \
  --host 0.0.0.0 \
  --port 1027 --dtype bfloat16

This fails w CUDA OOM though:

python -m sglang.launch_server \
  --model-path '/home/user/LLMs/weights/GLM-5.3-Flash-AWQ-W4A16' \
  --kt-weight-path '/home/user/LLMs/weights/GLM-5.3-Flash-AWQ-W4A16' \
  --kt-method RAWINT4 \
  --kt-cpuinfer 48 \
  --kt-threadpool-count 8 \
  --sleep-on-idle \
  --kt-num-gpu-experts 1 \
  --kt-gpu-prefill-token-threshold 2048 \
  --kt-enable-dynamic-expert-update \
  --kt-expert-placement-strategy uniform \
  --enable-p2p-check \
  --enable-mixed-chunk \
  --watchdog-timeout 3000 \
  --tp-size 4 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --kv-cache-dtype auto \
  --max-total-tokens 32768 \
  --chunked-prefill-size 2048 \
  --max-running-requests 1 \
  --attention-backend triton \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --served-model-name IO_model \
  --host 0.0.0.0 \
  --port 1027

Just have 4xA-100 and wanna serve GLM53Flash w docker ?
Here's your savior: image: lazymio/vllm-backport:latest-sm80
GitHub: https://github.com/wtdcode/vllm-backport

The huge GLM-5.2 works also with quantized CPU weights for me with this:

python -m sglang.launch_server \
  --model-path '/home/gele/LLMs/weights/GLM-5.2-FP16' \
  --kt-weight-path '/home/gele/LLMs/weights/GLM-5.2-FP16-AMXINT4-NUMA8' \
  --kt-cpuinfer 48 \
  --kt-threadpool-count 8 \
  --sleep-on-idle \
  --kt-num-gpu-experts 18 \
  --kt-method AMXINT4 \
  --kt-enable-dynamic-expert-update \
  --kt-expert-placement-strategy uniform \
  --disable-shared-experts-fusion \
  --enable-p2p-check \
  --enable-mixed-chunk \
  --watchdog-timeout 3000 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --mem-fraction-static 0.95 \
  --kv-cache-dtype auto \
  --max-total-tokens 224000 \
  --max-running-requests 4 \
  --cuda-graph-max-bs 4 \
  --attention-backend triton \
  --fp8-gemm-backend auto \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --served-model-name IO_model \
  --host 0.0.0.0 \
  --port 1027

@AGenchev commented on GitHub (Sep 19, 2026): In general, they support A-100. For example, this works on 4xA-100, 2x24-core AVX-512 CPUs: ```sh python -m sglang.launch_server \ --model-path '/home/user/LLMs/weights/GLM-5.2-FP16' \ --kt-weight-path '/home/user/LLMs/weights/GLM-5.2-FP16' \ --kt-cpuinfer 48 \ --kt-threadpool-count 8 \ --sleep-on-idle \ --kt-num-gpu-experts 15 \ --kt-method BF16 \ --kt-gpu-prefill-token-threshold 256 \ --kt-enable-dynamic-expert-update \ --kt-expert-placement-strategy uniform \ --enable-p2p-check \ --enable-mixed-chunk \ --watchdog-timeout 3000 \ --tp-size 4 \ --cuda-graph-max-bs 2 \ --trust-remote-code \ --mem-fraction-static 0.95 \ --kv-cache-dtype auto \ --max-total-tokens 192000 \ --max-running-requests 2 \ --attention-backend triton \ --fp8-gemm-backend auto \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --served-model-name IO_model \ --host 0.0.0.0 \ --port 1027 --dtype bfloat16 ``` This fails w CUDA OOM though: ```sh python -m sglang.launch_server \ --model-path '/home/user/LLMs/weights/GLM-5.3-Flash-AWQ-W4A16' \ --kt-weight-path '/home/user/LLMs/weights/GLM-5.3-Flash-AWQ-W4A16' \ --kt-method RAWINT4 \ --kt-cpuinfer 48 \ --kt-threadpool-count 8 \ --sleep-on-idle \ --kt-num-gpu-experts 1 \ --kt-gpu-prefill-token-threshold 2048 \ --kt-enable-dynamic-expert-update \ --kt-expert-placement-strategy uniform \ --enable-p2p-check \ --enable-mixed-chunk \ --watchdog-timeout 3000 \ --tp-size 4 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --kv-cache-dtype auto \ --max-total-tokens 32768 \ --chunked-prefill-size 2048 \ --max-running-requests 1 \ --attention-backend triton \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --served-model-name IO_model \ --host 0.0.0.0 \ --port 1027 ``` Just have 4xA-100 and wanna serve GLM53Flash w docker ? Here's your savior: image: lazymio/vllm-backport:latest-sm80 GitHub: https://github.com/wtdcode/vllm-backport The huge GLM-5.2 works also with quantized CPU weights for me with this: ```sh python -m sglang.launch_server \ --model-path '/home/gele/LLMs/weights/GLM-5.2-FP16' \ --kt-weight-path '/home/gele/LLMs/weights/GLM-5.2-FP16-AMXINT4-NUMA8' \ --kt-cpuinfer 48 \ --kt-threadpool-count 8 \ --sleep-on-idle \ --kt-num-gpu-experts 18 \ --kt-method AMXINT4 \ --kt-enable-dynamic-expert-update \ --kt-expert-placement-strategy uniform \ --disable-shared-experts-fusion \ --enable-p2p-check \ --enable-mixed-chunk \ --watchdog-timeout 3000 \ --tensor-parallel-size 4 \ --trust-remote-code \ --mem-fraction-static 0.95 \ --kv-cache-dtype auto \ --max-total-tokens 224000 \ --max-running-requests 4 \ --cuda-graph-max-bs 4 \ --attention-backend triton \ --fp8-gemm-backend auto \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --served-model-name IO_model \ --host 0.0.0.0 \ --port 1027 ```