[GH-ISSUE #1131] [Bug] Llama4-Scout OOM on 12GB card #789

New issue

Closed

opened 2026-10-29 14:44:31 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:44:31 +00:00

Owner

Originally created by @s60912frank on GitHub (Apr 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1131

Originally assigned to: @ovowei on GitHub.

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Llama4-Scout OOM on 12GB card (RTX3060)
Tutorial Said 10G VRAM would be enough.

Reproduction

Followed Tutorial.
And try to load smallest quant (Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf)
running with command:

python ktransformers/server/main.py \
  --port 8000 \
  --model_path /srv/home/workspace/ktransformers_test/meta-llama_Llama-4-Scout-17B-16E-Instruct \
  --gguf_path /srv/home/workspace/ktransformers_test/Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf \
  --optimize_config_path ktransformers/optimize/optimize_rules/Llama4-serve.yaml \
  --max_new_tokens 1024 \
  --cache_lens 32768 \
  --chunk_size 256 \
  --max_batch_size 4 \
  --backend_type balance_serve

Shows OOM (try to allocate 3.9GB more VRAM)

Environment

Dockefile used to build environment:

FROM pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel

ARG CPU_INSTRUCT=AVX2
WORKDIR /workspace
ENV CUDA_HOME=/usr/local/cuda
RUN apt update -y && \
    apt install -y --no-install-recommends \
    libtbb-dev \
    libssl-dev \
    libcurl4-openssl-dev \
    libaio1 \
    libaio-dev \
    libfmt-dev \
    libgflags-dev \
    zlib1g-dev \
    patchelf \
    git \
    wget \
    vim \
    gcc \
    g++ \
    cmake && \
    rm -rf /var/lib/apt/lists/*
RUN git clone -b support-llama4 https://github.com/kvcache-ai/ktransformers.git 

WORKDIR /workspace/ktransformers
RUN git submodule update --init --recursive
RUN pip install --upgrade pip
RUN pip install --no-cache-dir packaging ninja pyproject numpy cpufeature aiohttp zmq openai
RUN CPU_INSTRUCT=${CPU_INSTRUCT} \
    USE_BALANCE_SERVE=1 \
    KTRANSFORMERS_FORCE_BUILD=TRUE \
    TORCH_CUDA_ARCH_LIST="8.6+PTX" \
    bash ./install.sh && \
    pip install "transformers==4.51.0" && \
    pip cache purge

ENTRYPOINT ["tail", "-f", "/dev/null"]

Originally created by @s60912frank on GitHub (Apr 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1131 Originally assigned to: @ovowei on GitHub. ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug Llama4-Scout OOM on 12GB card (RTX3060) [Tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/llama4.md) Said 10G VRAM would be enough. ### Reproduction Followed [Tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/llama4.md). And try to load smallest quant (`Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf`) running with command: ``` python ktransformers/server/main.py \ --port 8000 \ --model_path /srv/home/workspace/ktransformers_test/meta-llama_Llama-4-Scout-17B-16E-Instruct \ --gguf_path /srv/home/workspace/ktransformers_test/Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf \ --optimize_config_path ktransformers/optimize/optimize_rules/Llama4-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --backend_type balance_serve ``` Shows OOM (try to allocate 3.9GB more VRAM) ### Environment Dockefile used to build environment: ``` FROM pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel ARG CPU_INSTRUCT=AVX2 WORKDIR /workspace ENV CUDA_HOME=/usr/local/cuda RUN apt update -y && \ apt install -y --no-install-recommends \ libtbb-dev \ libssl-dev \ libcurl4-openssl-dev \ libaio1 \ libaio-dev \ libfmt-dev \ libgflags-dev \ zlib1g-dev \ patchelf \ git \ wget \ vim \ gcc \ g++ \ cmake && \ rm -rf /var/lib/apt/lists/* RUN git clone -b support-llama4 https://github.com/kvcache-ai/ktransformers.git WORKDIR /workspace/ktransformers RUN git submodule update --init --recursive RUN pip install --upgrade pip RUN pip install --no-cache-dir packaging ninja pyproject numpy cpufeature aiohttp zmq openai RUN CPU_INSTRUCT=${CPU_INSTRUCT} \ USE_BALANCE_SERVE=1 \ KTRANSFORMERS_FORCE_BUILD=TRUE \ TORCH_CUDA_ARCH_LIST="8.6+PTX" \ bash ./install.sh && \ pip install "transformers==4.51.0" && \ pip cache purge ENTRYPOINT ["tail", "-f", "/dev/null"] ```

bsakaguchi closed this issue

2026-10-29 14:44:31 +00:00

bsakaguchi commented

2026-10-29 14:44:32 +00:00

Author

Owner

@ovowei commented on GitHub (Apr 15, 2025):

The 10GB mentioned in the tutorial refers to the runtime GPU memory usage. The peak memory usage can be higher, so a GPU with 12GB of memory might not be sufficient.

@ovowei commented on GitHub (Apr 15, 2025): The 10GB mentioned in the tutorial refers to the runtime GPU memory usage. The peak memory usage can be higher, so a GPU with 12GB of memory might not be sufficient.

bsakaguchi commented

2026-10-29 14:44:32 +00:00

Author

Owner

@s60912frank commented on GitHub (Apr 20, 2025):

Got it! I'll close this issue.

@s60912frank commented on GitHub (Apr 20, 2025): Got it! I'll close this issue.