[GH-ISSUE #151] Why generate "GGGGG...." ,when the input string is longer than a certain length in GGUF model? #132

New issue

Open

opened 2026-10-26 21:03:56 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:03:56 +00:00

Owner

Originally created by @hzgdeerHo on GitHub (Mar 26, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/151

TheBloke/deepseek-coder-33B-instruct-GGUF
deepseek-coder-33b-instruct.Q6_K.gguf
when I use llama cpp python to load the model , the model generates endless "GGGG...." ，But It works normally when the input question less than about 1000-2000 words.

   llm = Llama.from_pretrained(
        repo_id=args.model_name_or_path,
        # chat_format="llama-2",
        # chat_format="alpaca", 
        filename="deepseek-coder-33b-instruct.Q6_K.gguf",
        n_ctx=16000,            
        tokenizer=tokenizer,
         n_gpu_layers=-1,
         
        
        verbose=True
    )


    output = llm(
        user_prompt
       ,
        stream=True,
        max_tokens=12000,
        # max_new_tokens=4096,
        # do_sample=False, 
        top_k=50, 
        top_p=0.95,
        # num_return_sequences=1, 
        # eos_token_id=tokenizer.eos_token_id
        # repeat_penalty=1,
        
    )



    for chunk in output:
        delta = chunk['choices'][0]['text']
        print(delta,end='')

        bot_message+=delta

Originally created by @hzgdeerHo on GitHub (Mar 26, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/151 TheBloke/deepseek-coder-33B-instruct-GGUF deepseek-coder-33b-instruct.Q6_K.gguf when I use llama cpp python to load the model , the model generates endless "GGGG...." ，But It works normally when the input question less than about 1000-2000 words. llm = Llama.from_pretrained( repo_id=args.model_name_or_path, # chat_format="llama-2", # chat_format="alpaca", filename="deepseek-coder-33b-instruct.Q6_K.gguf", n_ctx=16000, tokenizer=tokenizer, n_gpu_layers=-1, verbose=True ) output = llm( user_prompt , stream=True, max_tokens=12000, # max_new_tokens=4096, # do_sample=False, top_k=50, top_p=0.95, # num_return_sequences=1, # eos_token_id=tokenizer.eos_token_id # repeat_penalty=1, ) for chunk in output: delta = chunk['choices'][0]['text'] print(delta,end='') bot_message+=delta

bsakaguchi commented

2026-10-26 21:03:57 +00:00

Author

Owner

@fubz commented on GitHub (Apr 4, 2024):

Just know you are not the only one that experiences this.

@fubz commented on GitHub (Apr 4, 2024): Just know you are not the only one that experiences this.

bsakaguchi commented

2026-10-26 21:03:57 +00:00

Author

Owner

@tastypear commented on GitHub (Jul 3, 2024):

have a try, turn on flash attention

@tastypear commented on GitHub (Jul 3, 2024): have a try, turn on flash attention

bsakaguchi commented

2026-10-26 21:03:57 +00:00

Author

Owner

@anrgct commented on GitHub (Jul 21, 2024):

I encountered the same issue when using LM Studio version 0.2.27. However, when I launched it using text-generation-webui, it worked normally and didn't produce the "GGGG" output even with long context.

@anrgct commented on GitHub (Jul 21, 2024): I encountered the same issue when using LM Studio version 0.2.27. However, when I launched it using text-generation-webui, it worked normally and didn't produce the "GGGG" output even with long context.