turboderp-org/exllamav3
1
0
Fork 0
mirror of https://github.com/turboderp-org/exllamav3.git synced 2026-07-12 10:07:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 1d44bd6279 gpt-oss: Add recurrent SWA mode (default) dev turboderp 2026-07-11 23:55:35 +02:00
  • 6312448a8f SlidingAttention: Add sink support turboderp 2026-07-11 23:55:22 +02:00
  • 1efa149bca Gemma4/TP: Add missing TP exports needed by Gemma4, enable TP support turboderp 2026-07-11 23:01:32 +02:00
  • b86ae91123 TP: Add TP boilerplate for SlidingAttention, add global checkpoint handle (to avoid possible collisions) turboderp 2026-07-11 23:00:47 +02:00
  • 7c8eb04cab gpt-oss: TODO item turboderp 2026-07-11 22:57:21 +02:00
  • b80d461993 BlockSparseMLP/TP: Allocate stable broadcast buffers for bsz1 graph turboderp 2026-07-11 21:22:30 +02:00
  • 2aa9b1603e TP: Use round-to-nearest for BF16 downcast during native all-reduce turboderp 2026-07-11 21:21:05 +02:00
  • 635bae660b BlockSparseMLP: Fix moe_bias_add_kernel expert indexing turboderp 2026-07-11 21:20:01 +02:00
  • 47c2d47df4 BlockSparseMLP: Add graph patching for run_bsz1 turboderp 2026-07-11 21:19:32 +02:00
  • 135f140bbc chat.py: Change /think for gpt-oss to mean "high" reasoning turboderp 2026-07-11 21:18:35 +02:00
  • 76cd9990aa chat.py: Formatting turboderp 2026-07-11 21:18:11 +02:00
  • f1a467bf6b gpt-oss: Add TP plumbing, enable TP turboderp 2026-07-11 18:43:42 +02:00
  • 010fa7f88f GatedRMSNorm: Initialize BC after module import turboderp 2026-07-11 16:52:44 +02:00
  • 8436248e01 TP: Fix NullConfig regression turboderp 2026-07-11 16:08:03 +02:00
  • e9d2c757ca Attention: Make sure quant-direct calls select from qc-aware backends preventing cache from dropping on the second chunk of a long prefill turboderp 2026-07-11 16:07:01 +02:00
  • 0aa4e48ea8 gpt-oss: Flag TP as unsupported (pending review) turboderp 2026-07-11 16:04:27 +02:00
  • afb59b826b chat.py: Increase default cache size and response length turboderp 2026-07-11 16:04:02 +02:00
  • 62598fc6a2 AsyncGenerator: Fix cancellation logic. turboderp 2026-07-11 12:12:45 +02:00
  • 72ed8485ca chat.py: Add /python command and basic bwrap sandbox turboderp 2026-07-11 02:14:34 +02:00
  • 9ca4e92368 chat.py: Add harmony console mode (gpt-oss) turboderp 2026-07-11 00:46:44 +02:00
  • 386c467647 convert.py: Make mul1 the default codebook for new models turboderp 2026-07-11 00:26:03 +02:00
  • 5abaac7779 DFlash: Support direct quantized cache updates without FP16 staging turboderp 2026-07-11 00:25:32 +02:00
  • 95d3bdee43 Attention/Cache: Allow non-causal-spans with quantized cache without FP16 staging, remove redundant placeholder while loading, fix kernel shmem overflow when K and V bitrate exceed 13 bits total turboderp 2026-07-11 00:24:51 +02:00
  • 74b3cec2bf requirements: Unpin pydantic, update setup.py install_requires turboderp 2026-07-10 23:17:09 +02:00
  • 1686a152b1 Merge pull request #235 from kumburovicbranko682-boop/contribai/improve/quality/undefined-variable-cache-size-causes-nam master turboderp 2026-07-10 22:44:53 +02:00
  • 3fd4ae7dc7 Merge pull request #236 from kumburovicbranko682-boop/contribai/improve/quality/bare-except-swallows-all-exceptions-and- turboderp 2026-07-10 22:44:14 +02:00
  • d8cd7fe84b Formatron: Monkey-patch the pydantic.typing namespace to restore missing aliases causing formatron to fail to import on recent pydantic versions turboderp 2026-07-10 22:31:11 +02:00
  • 1afffa690b Merge remote-tracking branch 'origin/dev' into dev turboderp 2026-07-10 22:05:57 +02:00
  • 65d3eb02c7 Merge pull request #240 from caipiralink/fix/gemma4-serving turboderp 2026-07-10 22:05:47 +02:00
  • 449ea5bd8d Gemma4: Correctly handle atomic prefill when multimodal token span is longer than the sliding attn window turboderp 2026-07-10 22:02:46 +02:00
  • bba588ceb4 Merge pull request #241 from caipiralink/fix/atomic-mm-recurrent-desync turboderp 2026-07-10 21:24:29 +02:00
  • 4857d9b57a Support gpt-oss, includes: expert bias, swiglu_oai activation fn, trimmed output in Attention and BlockSparseMLP (for padded hidden_size), attention sinks boilerplate turboderp 2026-07-10 19:53:11 +02:00
  • 721b164d2b Linear: Add MXFP4 dequant-on-load and padding (to support models with hidden_size not divisible by 128) turboderp 2026-07-10 19:41:49 +02:00
  • 6af187349e ppl.py: Set dequant mode for MXFP4 tensors in HF path, explicitly move input IDs to model device turboderp 2026-07-10 15:01:49 +02:00
  • e0bcd47704 RMSNorm: Add contiguous() guard before fused residual turboderp 2026-07-10 15:00:49 +02:00
  • 978b15ba79 safetensors: Recognize U8 dtype turboderp 2026-07-10 14:59:27 +02:00
  • a73786c930 RoPE: Support YaRN truncate argument turboderp 2026-07-10 14:58:56 +02:00
  • 4b7499616d Attn: Support learned attention sinks turboderp 2026-07-10 02:02:38 +02:00
  • b737db9322 DFlash: Remove dedicated DFlashAttention module (last place FA2 was required), specialize Attention instead, make FA2 an optional dependency turboderp 2026-07-10 00:42:57 +02:00
  • 7f0215d286 SlidingAttention: Add C++/graph path turboderp 2026-07-09 23:39:34 +02:00
  • 588e7c66ac SlidingAttention: Replace FA2 code path, update Triton attn kernel with SWA dual-KV-input mode turboderp 2026-07-09 22:56:36 +02:00
  • a82cafa740 Remove xformers requirement, respect ImportError as well as ModuleNotFoundError turboderp 2026-07-09 21:31:35 +02:00
  • 946c704c47 Add env_vars.md turboderp 2026-07-09 21:16:52 +02:00
  • f2240dcf6c Enable int8 mode by default (mul1 only), enable attn BC/graph by default turboderp 2026-07-09 21:14:19 +02:00
  • 52b47f0365 Attn: Fix excessive Triton recompiles over sequence length turboderp 2026-07-09 20:58:43 +02:00
  • 374ac60df7 Attn: Add C++/graph path for bsz <= 4, q_len <= 16 turboderp 2026-07-09 20:22:43 +02:00
  • 91404f4e5f MLP: Enable BC/graph path without fused up/gate proj (int8 mode) turboderp 2026-07-09 20:19:01 +02:00
  • 68d1eafe6a GEMV: Don't disable mgemm when int8 mode enabled and tensors use mcg codebook turboderp 2026-07-09 19:50:54 +02:00
  • c20c1fdb8c Graph: Add quant_cache_paged graph boilerplate turboderp 2026-07-09 19:16:38 +02:00
  • b15d0bb511 Graph: Add rope graph boilerplate turboderp 2026-07-09 19:15:47 +02:00
  • 0f3d34d12f Graph: Add CUDA driver API helpers and wrapper class for ahead-of-time compiled Triton kernels (cubins), extend graph wrapper with ability to patch Triton nodes via driver turboderp 2026-07-09 19:14:07 +02:00
  • 178984a59d Fix missing includes turboderp 2026-07-09 19:07:46 +02:00
  • 30c0c86a05 Graphs: Add rms_norm graph boilerplate turboderp 2026-07-09 19:07:15 +02:00
  • 2d5e2c5e23 Graphs: Add deinterleave_qg graph boilerplate turboderp 2026-07-09 19:06:34 +02:00
  • 69ca2a5c0a Graphs: Add mul_sigmoid graph boilerplate turboderp 2026-07-09 19:05:56 +02:00
  • e69b817b21 Cache/Attn: Fuse cache dequant into attn kernel, skip staging cache layer in temporary tensors when not needed, switch to hybrid bitfield/bitplane qcache storage turboderp 2026-07-08 22:31:40 +02:00
  • aa0f766d00 New Triton paged-attn kernel turboderp 2026-07-08 19:39:28 +02:00
  • 84c390052b Cache: More efficient quantized packing, add cubic compander to approximate Lloyd-Max, cache quant test scripts and simulated quantization turboderp 2026-07-08 16:38:00 +02:00
  • 485fa6c0c9 Add GEMV kernel turboderp 2026-07-08 02:48:32 +02:00
  • d0d21bb5c9 MoE routing: Improved warp reduce for topk, dedicated routing gemv kernel for bsz 1, fix hgemm before device guard turboderp 2026-07-07 21:22:34 +02:00
  • b95ac18189 Attn: Cache block tables for rectangular batches turboderp 2026-07-07 20:27:21 +02:00
  • b086c5b677 Attn: Cache dispatch backend selection per block turboderp 2026-07-07 20:26:52 +02:00
  • 19cb8f1922 Attn: Add custom kernel to de-interleave q, g tensors turboderp 2026-07-07 20:25:53 +02:00
  • 688d260750 recurrent_util: Add slot tensor cache turboderp 2026-07-07 20:24:48 +02:00
  • ddd9a9942c RMSNorm: Add fused residual-in kernel variant, adjust block size for low input dim turboderp 2026-07-07 19:41:08 +02:00
  • 30376042bf GatedDeltaNet: Add BC for Qwen3.5 split-tensor layout, graph boilerplate and code path for bsz 1, fused decode helper kernel, fused gemv kernel for B/A projections turboderp 2026-07-07 17:43:21 +02:00
  • f15cb4c334 GatedRMSNorm: Add graph boilerplate turboderp 2026-07-07 17:37:00 +02:00
  • f394db2bbe fix: sync recurrent state after atomic MM prefill caipira113 2026-07-08 18:20:51 +09:00
  • f3c8b75101 fix: guard formatron import against any error caipira113 2026-07-08 17:26:13 +09:00
  • cb26d959d7 fix: reshape non-contiguous attn output caipira113 2026-07-08 17:26:05 +09:00
  • 61323f3a0a conv1d: Add CUDA kernel turboderp 2026-07-07 16:11:43 +02:00
  • 9a1f0d003e Add mgemm_n_threshold in addition to mgemm_K_threshold for INT8 mode turboderp 2026-07-07 14:32:40 +02:00
  • 95a99dd1a0 Remove mgemv kernel entirely (no improvement measured) turboderp 2026-07-07 13:17:06 +02:00
  • 7b01fc550e Disable mgemv kernel, fix dangling workspace ptr, preserve TC matmul's shmem carveout in exl3_gemv_int8 turboderp 2026-07-07 04:09:20 +02:00
  • d5e4361fd5 MoE: Replace kernel round-robin assignment with dynamic ticket scheduler and add dynamic group sizing turboderp 2026-07-06 19:59:31 +02:00
  • e82c1cfe91 Split extension into mode compilation units turboderp 2026-07-06 19:57:25 +02:00
  • 2f297e496a GEMM/GEMV: Don't use mgemm by default for K=1..5 when int8 mode enabled turboderp 2026-07-06 02:59:15 +02:00
  • 9fe8b47138 MoE kernel: Add mul1 instances turboderp 2026-07-06 02:57:48 +02:00
  • 59e76bdcf0 GEMV: Add m=2 instance turboderp 2026-07-06 02:56:39 +02:00
  • 701656dd87 GEMV: Add experimental fused-int8 kernels turboderp 2026-07-05 14:22:22 +02:00
  • 8d9fd249c6 model_diff.py: Add per-token eval turboderp 2026-07-05 14:17:01 +02:00
  • 6faf89f292 convert_mtp.py: Use mul1 codebook by default for standalone MTP convert turboderp 2026-07-03 00:08:37 +02:00
  • a8012394be Quant: Optimize mul1 codebook, vabsdiff4 -> __dp4a turboderp 2026-07-03 00:08:08 +02:00
  • 2719af2c3b GEMM: Use fp16 MMA on Ampere (but still fold into fp32 accumulator) turboderp 2026-07-02 00:14:54 +02:00
  • edb2ac365f GEMM: Remove Ampere mul_const_u32 hack (seems no longer relevant on cu132) turboderp 2026-07-02 00:12:21 +02:00
  • c1815809dd model_init: Don't assume MTP args are present turboderp 2026-07-01 23:06:32 +02:00
  • 78b319e9bc RoPE: Accept float type for original_max_position_embeddings as long as the value is integer. Fix attn_factor when mscale and mscale_all_dim are present. turboderp 2026-07-01 00:10:48 +02:00
  • f35dedad5b chat.py: Add /svg command to extract, save and open SVG block from last model response turboderp 2026-07-01 00:08:04 +02:00
  • 70d5c8c7e5 refactor: bare except: swallows all exceptions and masks broken installs kumburovicbranko682-boop 2026-06-28 08:38:37 +08:00
  • 9816f9cdbd refactor: undefined variable cache_size causes nameerror at runtime kumburovicbranko682-boop 2026-06-28 08:36:24 +08:00
  • 188d073ed1 Ngram draft: Don't generate draft for jobs that are still in prefill stage turboderp 2026-06-26 20:49:38 +02:00
  • 24022f4d73 Qwen3.5-VL: Add qwen3_5_moe_vision alias turboderp 2026-06-26 19:52:30 +02:00
  • 55ebf029f2 Merge pull request #231 from Kira-PH/patch-1 turboderp 2026-06-21 12:47:08 +02:00
  • 4e88ece3ff Merge pull request #232 from Kira-PH/patch-2 turboderp 2026-06-21 12:43:02 +02:00
  • 544e12ca7f Merge pull request #234 from remichu-ai/fix/tp-gather-self-copy-offset turboderp 2026-06-21 12:40:58 +02:00
  • 5bcb6ed286 Fix TP gather self-copy offset remichu 2026-06-21 18:38:03 +08:00
  • 0953a739a6 TP: Apply correct offset in gather kernel when output device isn't first shard (issue #233) turboderp 2026-06-21 12:34:01 +02:00
  • 660d463ab8 Update pydantic version constraint in requirements Kira Phoenix-Hulme 2026-06-20 15:36:03 +01:00
  • 6a22c03a78 Fixed key_fused_qkvz and key_fused_ba being overwritten with None Kira Phoenix-Hulme 2026-06-20 15:23:56 +01:00
  • c5d9c65796 Bump to v0.0.43 v0.0.43 turboderp 2026-06-14 18:56:03 +02:00