ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-01-26 17:20:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,931 Commits 672 Branches 1 Tag
d894998fa6337f2bfe2bb2c41f11ceca79920b86
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
firecoperana d894998fa6 Add --webui arg to launch llama.cpp new webui (#786) 
* Add new webui from llama.cpp

* Add new webui

* feat: Improve mobile UI for Settings Dialog (#16084)

* feat: Improve mobile UI for Settings Dialog

* chore: update webui build output

* fix: Linting errors

* chore: update webui build output
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatSettings/ChatSettingsFields.svelte
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatSettings/ChatSettingsSection.svelte
#	tools/server/public/index.html.gz

* webui : fix handling incomplete chunks (#16107)

* Always show message actions for mobile UI + improvements for user message sizing (#16076)

# Conflicts:
#	.gitignore
#	examples/server/webui_llamacpp/package.json
#	examples/server/webui_llamacpp/scripts/dev.sh
#	tools/server/webui/scripts/post-build.sh

* webui: switch to hash-based routing (alternative of #16079) (#16157)

* Switched web UI to hash-based routing

* Added hash to missed goto function call

* Removed outdated SPA handling code

* Fixed broken sidebar home link
# Conflicts:
#	examples/server/webui_llamacpp/src/routes/+layout.ts
#	tools/server/server.cpp

* Allow viewing conversations even when llama server is down (#16255)

* webui: allow viewing conversations and sending messages even if llama-server is down

- Cached llama.cpp server properties in browser localStorage on startup, persisting successful fetches and reloading them when refresh attempts fail so the chat UI continues to render while the backend is unavailable.
- Cleared the stored server properties when resetting the store to prevent stale capability data after cache-backed operation.
- Kept the original error-splash behavior when no cached props exist so fresh installs still surface a clear failure state instead of rendering stale data.

* feat: Add UI for `props` endpoint unavailable + cleanup logic

* webui: extend cached props fallback to offline errors

Treat connection failures (refused, DNS, timeout, fetch) the same way as
server 5xx so the warning banner shows up when cache is available, instead
of falling back to a full error screen.

* webui: Left the chat form enabled when a server warning is present so operators can keep sending messages

e.g., to restart the backend over llama-swap, even while cached /props data is in use

* chore: update webui build output

---------

Co-authored-by: Pascal <admin@serveurperso.com>
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatScreen/ChatScreenWarning.svelte
#	examples/server/webui_llamacpp/src/lib/constants/localstorage-keys.ts

* Enhance text file detection logic for file attachments (#16199)

* feat: Enhances text file detection logic

* chore: Build static `webui` output

* chore: update webui build output
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/constants/binary-detection.ts

* Show message actions by default (#16289)

* fix: preserved zero values in chat settings inputs and textareas by switching to nullish coalescing for field values and default placeholders (#16312)

* Improve Mobile UI for dialogs and action dropdowns (#16222)

* fix: Always show conversation item actions

* feat: Improve Alert Dialog and Dialog mobile UI

* feat: Add settings reset to default confirmation

* fix: Close Edit dialog on save

* chore: update webui build output

* webui: implement proper z-index system and scroll management

- Add CSS variable for centralized z-index control
- Fix dropdown positioning with Settings dialog conflicts
- Prevent external scroll interference with proper event handling
- Clean up hardcoded z-index values for maintainable architecture

* webui: ensured the settings dialog enforces dynamic viewport height on mobile while retaining existing desktop sizing overrides

* feat: Use `dvh` instead of computed px height for dialogs max height on mobile

* chore: update webui build output

* feat: Improve Settings fields UI

* chore: update webui build output

* chore: update webui build output

---------

Co-authored-by: Pascal <admin@serveurperso.com>

* Fix thinking blocks with quotes + add handling `[THINK]...[/THINK]` blocks (#16326)

* fix: prevent reasoning blocks with quotes from being truncated

* chore: update webui build output

* feat: Improve thinking content parsing

* test: Adds ChatMessage component stories for different thinking blocks

* chore: update webui build output

* fix: ChatMessage story fix

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Chatapi ignore empty sampling (#16330)

* fix: skip empty sampling fields instead of coercing to 0 in chat API options

* chore: update webui build output

* webui: Remove running `llama-server` within WebUI `dev.sh` script (#16363)

* Add optional setting for showing "Model used:" information (#16337)

* feat: Add a setting to include model name used to generate the message

* feat: UI improvements

* feat: Save model info along with the database message entry creation

* chore: Build webui static output

* Improve code block color theming (#16325)

* feat: Improve code block theming

* chore: update webui build output

* chore: Update webui static build

* Conversation action dialogs as singletons from Chat Sidebar + apply conditional rendering for Actions Dropdown for Chat Conversation Items (#16369)

* fix: Render Conversation action dialogs as singletons from Chat Sidebar level

* chore: update webui build output

* fix: Render Actions Dropdown conditionally only when user hovers conversation item + remove unused markup

* chore: Update webui static build

* fix: Always truncate conversation names

* chore: Update webui static build

* fix: track viewportHeight via window.innerHeight to avoid unwanted scrolling (#16356)

Use <svelte:window bind:innerHeight> instead of manual resize listener

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui : Fix messages payload sent to chat completions (#16402)

* fix: Include just the currently active message branches instead of all in chat completions request

* chore: Build webui static output

* chore: Formatting

* chore: update webui build output

* Capture model name only after first token (streaming) or completed request (#16405)

* feat: Capture model name only after first token (streaming) or completed request (non-streaming)

* chore: update webui build output

* chore: update webui build output

* Fix missing messages on sibling navigation (#16408)

* fix: resolve message disappearing issue when navigating between regenerated siblings by using current leaf nodes instead of cached sibling IDs

* chore: update webui build output

* chore: update webui build output

* webui : added download action (#13552) (#16282)

* webui : added download action (#13552)

* webui : import and export (for all conversations)

* webui : fixed download-format, import of one conversation

* webui : add ExportedConversations type for chat import/export

* feat: Update naming & order

* chore: Linting

* webui : Updated static build output

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* refactor: centralize CoT parsing in backend for streaming mode (#16394)

* refactor: unify reasoning handling via backend reasoning_content, drop frontend tag parsing

- Updated the chat message component to surface backend-supplied reasoning via message.thinking while showing the raw assistant content without inline tag scrubbing
- Simplified chat streaming to append content chunks directly, stream reasoning into the message model, and persist any partial reasoning when generation stops
- Refactored the chat service SSE handler to rely on server-provided reasoning_content, removing legacy <think> parsing logic
- Refreshed Storybook data and streaming flows to populate the thinking field explicitly for static and streaming assistant messages

* refactor: implement streaming-aware universal reasoning parser

Remove the streaming mode limitation from --reasoning-format by refactoring
try_parse_reasoning() to handle incremental parsing of <think> tags across
all formats.

- Rework try_parse_reasoning() to track whitespace, partial tags, and
  multiple reasoning segments, allowing proper separation of reasoning_content
  and content in streaming mode
- Parse reasoning tags before tool call handling in content-only and Llama 3.x
  formats to ensure inline <think> blocks are captured correctly
- Change default reasoning_format from 'auto' to 'deepseek' for consistent
  behavior
- Add 'deepseek-legacy' option to preserve old inline behavior when needed
- Update CLI help and documentation to reflect streaming support
- Add parser tests for inline <think>...</think> segments

The parser now continues processing content after </think> closes instead of
stopping, enabling proper message.reasoning_content and message.content
separation in both streaming and non-streaming modes.

Fixes the issue where streaming responses would dump everything (including
post-thinking content) into reasoning_content while leaving content empty.

* refactor: address review feedback from allozaur

- Passed the assistant message content directly to ChatMessageAssistant to drop the redundant derived state in the chat message component
- Simplified chat streaming updates by removing unused partial-thinking handling and persisting partial responses straight from currentResponse
- Refreshed the ChatMessage stories to cover standard and reasoning scenarios without the old THINK-tag parsing examples

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* refactor: restore forced reasoning prefix to pass test-chat ([chat] All tests passed)

- store the exact sequence seen on input when 'thinking_forced_open' enforces a reasoning block
- inject this prefix before the first accumulated segment in 'reasoning_content', then clear it to avoid duplication
- repeat the capture on every new 'start_think' detection to properly handle partial/streaming flows

* refactor: address review feedback from ngxson

* debug: say goodbye to curl -N, hello one-click raw stream

- adds a new checkbox in the WebUI to display raw LLM output without backend parsing or frontend Markdown rendering

* Update tools/server/webui/src/lib/components/app/chat/ChatMessages/ChatMessage.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui: add Storybook example for raw LLM output and scope reasoning format toggle per story

- Added a Storybook example that showcases the chat message component in raw LLM output mode with the provided trace sample
- Updated every ChatMessage story to toggle the disableReasoningFormat setting so the raw-output rendering remains scoped to its own example

* npm run format

* chat-parser: address review feedback from ngxson

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
# Conflicts:
#	common/arg.cpp
#	examples/server/webui_llamacpp/src/lib/utils/thinking.ts
#	tools/server/README.md

* No markdown in cot (#16483)

* fix: let the model think in plaintext

* chore: npm run format + npm run build

* webui: updated the chat service to only include max_tokens in the req… (#16489)

* webui: updated the chat service to only include max_tokens in the request payload when the setting is explicitly provided, while still mapping explicit zero or null values to the infinite-token sentinel

* chore: update webui build output

* feat: render user content as markdown option (#16358)

* feat: render user content as markdown option
- Add a persisted 'renderUserContentAsMarkdown' preference to the settings defaults and info metadata so the choice survives reloads like other options
- Surface the new 'Render user content as Markdown' checkbox in the General section of the chat settings dialog, beneath the PDF toggle
- Render user chat messages with 'MarkdownContent' when the new setting is enabled, matching assistant formatting while preserving the existing card styling otherwise
- chore: update webui build output

* chore: update webui build output

* webui: remove client-side context pre-check and rely on backend for limits (#16506)

* fix: make SSE client robust to premature [DONE] in agentic proxy chains

* webui: remove client-side context pre-check and rely on backend for limits

Removed the client-side context window pre-check and now simply sends messages
while keeping the dialog imports limited to core components, eliminating the
maximum context alert path

Simplified streaming and non-streaming chat error handling to surface a generic
'No response received from server' error whenever the backend returns no content

Removed the obsolete maxContextError plumbing from the chat store so state
management now focuses on the core message flow without special context-limit cases

* webui: cosmetic rename of error messages

* Update tools/server/webui/src/lib/stores/chat.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/stores/chat.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* chore: update webui build output

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/dialogs/ChatErrorDialog.svelte
#	examples/server/webui_llamacpp/src/lib/components/app/dialogs/MaximumContextAlertDialog.svelte
#	examples/server/webui_llamacpp/src/lib/services/context.ts

* fix: add remark plugin to render raw HTML as literal text (#16505)

* fix: add remark plugin to render raw HTML as literal text

Implemented a missing MDAST stage to neutralize raw HTML like major LLM WebUIs
do ensuring consistent and safe Markdown rendering

Introduced 'remarkLiteralHtml', a plugin that converts raw HTML nodes in the
Markdown AST into plain-text equivalents while preserving indentation and
line breaks. This ensures consistent rendering and prevents unintended HTML
execution, without altering valid Markdown structure

Kept 'remarkRehype' in the pipeline since it performs the required conversion
from MDAST to HAST for KaTeX, syntax highlighting, and HTML serialization

Refined the link-enhancement logic to skip unnecessary DOM rewrites,
fixing a subtle bug where extra paragraphs were injected after the first
line due to full innerHTML reconstruction, and ensuring links open in new
tabs only when required

Final pipeline: remarkGfm -> remarkMath -> remarkBreaks -> remarkLiteralHtml
-> remarkRehype -> rehypeKatex -> rehypeHighlight -> rehypeStringify

* fix: address review feedback from allozaur

* chore: update webui build output
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/constants/literal-html.ts

* Add server-driven parameter defaults and syncing (#16515)

# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatSettings/ParameterSourceIndicator.svelte
#	examples/server/webui_llamacpp/src/lib/constants/precision.ts
#	examples/server/webui_llamacpp/src/lib/services/parameter-sync.spec.ts
#	examples/server/webui_llamacpp/src/lib/services/parameter-sync.ts
#	examples/server/webui_llamacpp/src/lib/utils/config-helpers.ts
#	examples/server/webui_llamacpp/src/lib/utils/precision.ts

* fix: added a normalization step for MathJax-style \[\] and \(\) delimiters (#16599)

* fix: added a normalization step for MathJax-style \[\] and \(\) delimiters

So inline and block equations are converted before KaTeX rendering,
enabling proper display of model-generated LaTeX in the WebUI

* chore: update webui build output

* webui: reorganize settings layout (#16607)

* webui: reorganize settings layout

* chore: update webui build output

* fix: remove unused variable

* chore: update webui build output

* Enable per-conversation loading states to allow having parallel conversations (#16327)

* feat: Per-conversation loading states and tracking streaming stats

* chore: update webui build output

* refactor: Chat state management

Consolidates loading state management by using a global `isLoading` store synchronized with individual conversation states.

This change ensures proper reactivity and avoids potential race conditions when updating the UI based on the loading status of different conversations. It also improves the accuracy of statistics displayed.

Additionally, slots service methods are updated to use conversation IDs for per-conversation state management, avoiding global state pollution.

* feat: Adds loading indicator to conversation items

* chore: update webui build output

* fix: Fix aborting chat streaming

Improves the chat stream abortion process by ensuring that partial responses are saved before the abort signal is sent.

This avoids a race condition where the onError callback could clear the streaming state before the partial response is saved. Additionally, the stream reading loop and callbacks are now checked for abort signals to prevent further processing after abortion.

* refactor: Remove redundant comments

* chore: build webui static output

* refactor: Cleanup

* chore: update webui build output

* chore: update webui build output

* fix: Conversation loading indicator for regenerating messages

* chore: update webui static build

* feat: Improve configuration

* feat: Install `http-server` as dev dependency to not need to rely on `npx` in CI

* Import/Export UX improvements (#16619)

* webui : added download action (#13552)

* webui : import and export (for all conversations)

* webui : fixed download-format, import of one conversation

* webui : add ExportedConversations type for chat import/export

* feat: Update naming & order

* chore: Linting

* feat: Import/Export UX improvements

* chore: update webui build output

* feat: Update UI placement of Import/Export tab in Chat Settings Dialog

* refactor: Cleanup

chore: update webui build output

* feat: Enable shift-click multiple conversation items selection

* chore: update webui static build

* chore: update webui static build

---------

Co-authored-by: Sascha Rogmann <github@rogmann.org>
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatSettings/ConversationSelectionDialog.svelte
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatSettings/ImportExportTab.svelte
#	examples/server/webui_llamacpp/src/lib/utils/conversation-utils.ts

* Prevent premature submission on IME input (#16673)

* fix: Prevent premature submission on IME input

* chore: update webui static build

* refactor: Put IME completion checker in a helper function and add checking for `KeyboardEvent.eventKey === 229`

* chore: update webui static build

* chore: update webui static build

* chore: update webui static build
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/utils/is-ime-composing.ts

* Handle legacy 'context' attachments (#16687)

* webui: introduce OpenAI-compatible model selector in JSON payload (#16562)

* webui: introduce OpenAI-compatible model selector in JSON payload

* webui: restore OpenAI-Compatible model source of truth and unify metadata capture

This change re-establishes a single, reliable source of truth for the active model:
fully aligned with the OpenAI-Compat API behavior

It introduces a unified metadata flow that captures the model field from both
streaming and non-streaming responses, wiring a new onModel callback through ChatService
The model name is now resolved directly from the API payload rather than relying on
server /props or UI assumptions

ChatStore records and persists the resolved model for each assistant message during
streaming, ensuring consistency across the UI and database
Type definitions for API and settings were also extended to include model metadata
and the onModel callback, completing the alignment with OpenAI-Compat semantics

* webui: address review feedback from allozaur

* webui: move model selector into ChatForm (idea by @allozaur)

* webui: make model selector more subtle and integrated into ChatForm

* webui: replaced the Flowbite selector with a native Svelte dropdown

* webui: add developer setting to toggle the chat model selector

* webui: address review feedback from allozaur

Normalized streamed model names during chat updates
by trimming input and removing directory components before saving
or persisting them, so the conversation UI shows only the filename

Forced model names within the chat form selector dropdown to render as
a single-line, truncated entry with a tooltip revealing the full name

* webui: toggle displayed model source for legacy vs OpenAI-Compat modes

When the selector is disabled, it falls back to the active server model name from /props

When the model selector is enabled, the displayed model comes from the message metadata
(the one explicitly selected and sent in the request)

* Update tools/server/webui/src/lib/components/app/chat/ChatForm/ChatFormActions.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/constants/localstorage-keys.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/components/app/chat/ChatForm/ChatFormModelSelector.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/components/app/chat/ChatMessages/ChatMessageAssistant.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/services/chat.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/services/chat.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui: refactor model selector and persistence helpers

- Replace inline portal and event listeners with proper Svelte bindings
- Introduce 'persisted' store helper for localStorage sync without runes
- Extract 'normalizeModelName' utils + Vitest coverage
- Simplify ChatFormModelSelector structure and cleanup logic

Replaced the persisted store helper's use of '$state/$effect' runes with
a plain TS implementation to prevent orphaned effect runtime errors
outside component context

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui: document normalizeModelName usage with inline examples

* Update tools/server/webui/src/lib/components/app/chat/ChatForm/ChatFormModelSelector.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/stores/models.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* Update tools/server/webui/src/lib/stores/models.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui: extract ModelOption type into dedicated models.d.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* webui: refine ChatMessageAssistant displayedModel source logic

* webui: stabilize dropdown, simplify model extraction, and init assistant model field

* chore: update webui static build

* Update tools/server/webui/src/lib/components/app/chat/ChatMessages/ChatMessageAssistant.svelte

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* chore: npm format, update webui static build

* webui: align sidebar trigger position, remove z-index glitch

* chore: update webui build output

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
# Conflicts:
#	examples/server/webui_llamacpp/src/lib/components/app/chat/ChatForm/ChatFormModelSelector.svelte
#	examples/server/webui_llamacpp/src/lib/services/models.ts
#	examples/server/webui_llamacpp/src/lib/stores/models.svelte.ts
#	examples/server/webui_llamacpp/src/lib/stores/persisted.svelte.ts
#	examples/server/webui_llamacpp/src/lib/types/models.d.ts
#	examples/server/webui_llamacpp/src/lib/utils/model-names.test.ts
#	examples/server/webui_llamacpp/src/lib/utils/model-names.ts
#	examples/server/webui_llamacpp/src/lib/utils/portal-to-body.ts

* webui: support q URL parameter (#16728)

* webui: support q URL parameter

Fixes #16722
I’ve checked that it works with Firefox’s AI tools

* webui: apply suggestions from code review

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* chore: update webui static build

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

* build fix

---------

Co-authored-by: firecoperana <firecoperana>
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
Co-authored-by: Quentin Bramas <quentin.bramas@gmail.com>
Co-authored-by: Isaac McFadyen <isaac@imcf.me>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sascha Rogmann <59577610+srogmann@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
Co-authored-by: Sascha Rogmann <github@rogmann.org>
Co-authored-by: Florian Badie <florianbadie@odrling.xyz>
2025-10-27 14:22:02 +02:00
.devops
Merge mainline - Aug 12 2024 (#17)
2024-08-12 15:14:32 +02:00
.github
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
ci
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
cmake
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
common
Add --webui arg to launch llama.cpp new webui (#786)
2025-10-27 14:22:02 +02:00
docs
Tool calls support from mainline (#723)
2025-09-01 08:38:49 +03:00
examples
Add --webui arg to launch llama.cpp new webui (#786)
2025-10-27 14:22:02 +02:00
ggml
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
gguf-py
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
github-data
Add GitHub data: filename sanitization (#640)
2025-07-23 13:31:53 +02:00
grammars
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
include
Fused mul + multi_add op (#858)
2025-10-24 07:40:35 +03:00
media
README: add graphic for matrix multiplication (#6881)
2024-04-24 21:29:13 +02:00
models
Deepseek V3.1 native tool calling support (OpenAI Style) (#771)
2025-09-13 07:51:40 +02:00
pocs
build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)
2024-06-13 00:41:52 +01:00
prompts
llama : add Qwen support (#4281)
2023-12-01 20:16:31 +02:00
requirements
Tool calls support from mainline (#723)
2025-09-01 08:38:49 +03:00
scripts
sync: vendor (#799)
2025-09-26 18:22:47 +02:00
spm-headers
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
src
Change flash attention and fmoe to be on by default (#863)
2025-10-25 09:37:28 +03:00
tests
Add --webui arg to launch llama.cpp new webui (#786)
2025-10-27 14:22:02 +02:00
vendor
Move minja and nlohmann/json to vendor (#802)
2025-09-27 09:12:35 +02:00
.dockerignore
build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)
2024-06-13 00:41:52 +01:00
.ecrc
Nomic Vulkan backend (#4456)
2024-01-29 15:50:50 -05:00
.editorconfig
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
.flake8
py : logging and flake8 suppression refactoring (#7081)
2024-05-05 08:07:48 +03:00
.gitignore
Add --webui arg to launch llama.cpp new webui (#786)
2025-10-27 14:22:02 +02:00
.gitmodules
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
.mailmap
Add .mailmap
2025-07-22 14:53:50 +03:00
.pre-commit-config.yaml
convert.py : add python logging instead of print() (#6511)
2024-05-03 22:36:41 +03:00
AUTHORS
Update AUTHORS
2025-07-23 18:14:51 +02:00
CMakeLists.txt
Tool calls support from mainline (#723)
2025-09-01 08:38:49 +03:00
CMakePresets.json
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
CONTRIBUTING.md
Merge mainline - Aug 12 2024 (#17)
2024-08-12 15:14:32 +02:00
convert_hf_to_gguf_update.py
model : add grok-2 support (#782)
2025-09-23 16:31:01 +02:00
convert_hf_to_gguf.py
Ling-1T convert fixup (#837)
2025-10-17 07:52:31 +03:00
convert_llama_ggml_to_gguf.py
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
convert_lora_to_gguf.py
Fix missing rope_freqs with convert_hf_to_gguf (#402)
2025-05-09 09:17:41 -05:00
flake.lock
Merge mainline - Aug 12 2024 (#17)
2024-08-12 15:14:32 +02:00
flake.nix
build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)
2024-06-13 00:41:52 +01:00
LICENSE
Use links for ggml/llama.cpp authors (#318)
2025-04-07 17:25:06 +02:00
Makefile
Function calling support for Kimi-K2 (#628)
2025-07-23 18:11:42 +02:00
mypy.ini
convert : partially revert PR #4818 (#5041)
2024-01-20 18:14:18 -05:00
Package.swift
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
poetry.lock
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pyproject.toml
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pyrightconfig.json
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
README.md
Update README.md
2025-07-23 19:38:54 +02:00
requirements.txt
Tool calls support from mainline (#723)
2025-09-01 08:38:49 +03:00
test-function-calls.md
Function calling support for Kimi-K2 (#628)
2025-07-23 18:11:42 +02:00

README.md

ik_llama.cpp: llama.cpp fork with better CPU performance

License: MIT

TL;DR

This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU/CPU inference, row-interleaved quant packing, etc.

Latest News

Model Support

LlaMA-3-Nemotron PR 377, Qwen3 PR 355, GLM-4 PR 344, Command-A PR 341, bitnet-b1.58-2B-4T PR 337, LLaMA-4 PR 321, Gemma3 PR 276, DeepSeek-V3 PR 176, Kimi-2 PR 609, dots.llm1 PR 573, Hunyuan PR 565

Quantization

Quantization additions

Trellis quants (IQ1_KT, IQ2_KT, IQ3_KT, IQ4_KT)

Information and the original CUDA implementation in PR 113. Additional implementations: Metal PR 475, Neon PR 471, CPU PR 441. IQ1_KT was added more recently in PR 616. Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see PR 529 and PRs quoted there for details.

IQK quants

Information can be found in Discussion 8.

Initial implementations (Zen4, AVX2, NEON): IQ5_KS_R4 PR 426, IQ5_KS PR 422, IQ4_KS_R4 PR 150, IQ5_K_R4 PR 149, IQ2_K_R4 PR 146, IQ3_K_R4 PR 145, IQ4_K_R4 PR 138, IQ4_KSS PR 89, IQ2_KS PR 85, IQ4_KS PR 83, IQ6_K PR 14, IQ2_K, IQ3_K and IQ5_K PR 7, IQ4_K PR 6

Cuda implementations: IQ4_KS_R4 and IQ5_KS_R4 PR 493, IQ1_S_R4 PR 492, IQ1_M_R4 PR 494. IQ4_KS_R4 and IQ5_KS_R4 PR 462, IQ2_K_R4, IQ3_K_R4, IQ4_K_R4, IQ5_K_R4 PR 461, IQ4_K, IQ5_K, IQ6_K PR 417, IQ2_KS, IQ2_K, IQ3_K PR 418

IQ2_KL is a more recent addition in PR 602

Quantization improvements

IQ1_M PR 327, IQ2_XS PR 312, Q2_K, Q4_K, Q5_K, Q4_1, Q5_1 PR 302, Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NL PR 295

Quantization performance improvements

  • Much faster CPU prompt processing for all non-interleaved quants. Initial idea in PR 515 and PR 531, with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
  • All quantization types now have quantized matrix multiplication CUDA kernels, see PR 557 and several others
  • Faster CPU prompt processing for Trellis quants and MoE models. PR 488
  • Trellis quants: faster CPU prompt processing PR 482.
  • Minor (~2%) iq2_ks TG performance improvement on CUDA PR 468
  • Faster IQ3_KT and IQ4_KT PR 453
  • Zen4: Faster PP for IQ2_KS, IQ4_KS, IQ5_KS PR 428
  • Fast GEMM/GEMV for IQ1_S PR 212

Features

  • Function call support PR 628
  • Webui: New Features for Conversations, Settings, and Chat Messages PR 618
  • Legacy quants conversion schemes in convert_hf_to_gguf.py PR 449, Q6_0 in PR 483
  • June 8 2025: Webui updated (legacy still available when --path ./examples/server/public_legacy is passed) PR 481
  • June 8 2025: RPC improvements PR 480
  • June 7 2025: Add an endpoint that lists all the saved prompt caches to server PR 502
  • June 6 2025: Make prompt cache saving and restoring MLA aware PR 497
  • June 3 2025: Added samplers, XTC PR 486, top-n σ PR 489.
  • May 22 2025: Refactor iqk_mul_mat.cpp which speeds up compilation time significantly. PR 435
  • May 17 2025: Option to enable or disable the CPU FA kernels PR 429.
  • May 12 2025: User can now control if/which operations with tensors held in RAM are offloaded to the GPU. See PR 405
  • May 12 2025: Compatibility issues with mainline llama.cpp GGUFs for DeepSeek models with MLA enabled were resolved in PR 394. The lower prompt processing performance resulting from using llama.cpp-style MLA GGUFs was recovered in PR 409.
  • April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see PR 336
  • March 1 2025: Smart Expert Reduction for faster DeepSeek inference PR 239
  • Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) PR 232
  • Feb 23 2025: sweep-bench - better performance benchmarking PR 225
  • Feb 19 2025: Q8_KV - new type for 8-bit KV-cache quantization PR 208
  • March 7 2025: Custom quantization mixes using regular expressions PR 244

Performance improvements

  • Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see PR 520
  • May 13 2025: Better CPU FA performance for DeepSeek-Lite. PR 410
  • May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. PR 408
  • May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. PR 370
  • April 17 2025: Better CPU Flash Attention token generation performance. PR 332
  • April 3 2025: Much faster MoE implementation on Metal. PR 307
  • March 25 2025: Better MoE performance on CUDA PR 283
  • March 23 2025: Better batched processing speed for DeepSeek models PR 282
  • March 18 2025: Reduce compute buffer size PR 237
  • March 10 2025: Better TG performance for MoE models on CUDA PR 248
  • Feb 23 2025: Fused FFN ops for faster MoE inference PR 229

Flash-MLA

  • May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. PR 386. Caveat: Ampere or newer Nvidia GPU required
  • March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models PR 273
  • March 17 2025: 🚀 FlashMLA-2 performance improvements PR 253
  • March 12 2025: Allow Q8_0 KV cache with FlashMLA-2 on CUDA PR 265
  • March 9 2025: 🚀 FlashMLA on CUDA PR 247
  • March 8 2025: 🚀 Faster FlashMLA CPU implementation PR 243
  • March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention PR 240
  • Feb 27 2025: MLA without transposed cache PR 235
  • Feb 13 2025: Allow Q8_0 quantized cache with MLA PR 206
  • Feb 11 2025: 🚀 Flash Attention support for DeepSeek models PR 200
  • Feb 9 2025: 🚀 MLA for DeepSeek models PR 188

Fixes

  • Fix bug in MMVQ kernel PR 446
  • Fix AVX2 implementation of IQ4_K, IQ4_KS, IQ5_K, IQ6_K PR 427
  • Fix standard attention on the CPU PR 421
  • Fix imatrix calculation for MLA models PR 411
  • Fix new CUDA FA on Touring PR 413
  • Fix SER. CPU: PR 415 CUDA: PR 416

Resources

There is no single point of reference describing all new ik_llama.cpp features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition

  • The Wiki page has performance comparisons to mainline llama.cpp
  • This guide is a good place to start if you came here because of DeepSeek models
  • This discussion is about running DeepSeek-V3/R1 on a 16 x 3090 setup
  • This discussion describes the new quantization types available in ik_llama.cpp

Testing

Function Calls Tests

To run the function calls test suite:

cd build
cmake --build . --target test-function-calls
./bin/test-function-calls

The test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.

Contributing

Contributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.

License

MIT

Description
llama.cpp fork with additional SOTA quants and improved performance
Readme MIT 158 MiB
Languages
C++ 55.4%
C 16.4%
Cuda 14%
Python 5.5%
Metal 3%
Other 5.6%