[PR #96] [MERGED] Fix Kimi tokenizer and CUDA payload loading #96

New issue

Closed

opened 2026-10-29 21:07:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/96
Author: @yyj6666667
Created: 9/12/2026
Status: ✅ Merged
Merged: 9/13/2026
Merged by: @yyj6666667

Base: main ← Head: fix/kimi-tiktoken-guidance-20260912

📝 Commits (5)

2e46727 fix(kimi): initialize llguidance from native tiktoken encoding
b89594a test(kimi): follow repository import ordering
634b05e fix(packaging): preserve wheel-relative CUDA payload paths
d9749d3 style(kimi): format tokenizer factory call for CI
95eae11 test: focus Kimi tokenizer and payload layout coverage

📊 Changes

4 files changed (+102 additions, -3 deletions)

View changed files

📝 python/sglang/srt/constrained/llguidance_backend.py (+20 -1)
📝 sgl-kernel/python/sgl_kernel/payload_runtime.py (+4 -2)
➕ test/registered/unit/test_llguidance_kimi_tokenizer.py (+50 -0)
➕ test/registered/unit/test_payload_runtime_layout.py (+28 -0)

📄 Description

What does this PR do?

Use llguidance's native tiktoken support for Kimi tokenizers; other models keep the existing dispatch.
Resolve lazy native binaries using their wheel-relative paths, preserving bundled library lookup. Legacy manifests remain supported.

Two production files, +24/-3 lines, with two focused test files. The packaging counterpart is kvcache-ai/ktransformers#2204.

Tests

Focused tokenizer/payload tests: 6 passed in a fresh wheel-only environment.
Original Kimi tokenizer: multilingual token IDs, EOS, vocabulary, and JSON grammar acceptance/rejection passed.
Fresh native wheels: dependency / architecture audit and eight concurrent cold imports passed.
Clean training/serving installation passed. Qwen two-GPU, Kimi four-GPU, and public FP8 GLM-5.3-Flash eight-GPU generation and JSON-schema checks passed.
Kimi loaded both ordinary and expert LoRA from a freshly resumed training checkpoint; all converted tensors passed exact comparison. Full Neko style validation remains pending.

No model weights, attention kernels, or training runtime changes. Not a release sign-off.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/96 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 9/12/2026 **Status:** ✅ Merged **Merged:** 9/13/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `fix/kimi-tiktoken-guidance-20260912` --- ### 📝 Commits (5) - [`2e46727`](https://github.com/kvcache-ai/sglang/commit/2e46727a4e6de5a2d41c3fd69f1b9ca4cd774023) fix(kimi): initialize llguidance from native tiktoken encoding - [`b89594a`](https://github.com/kvcache-ai/sglang/commit/b89594ad81f6eb06bfb3c64ec0e419ab90b29cda) test(kimi): follow repository import ordering - [`634b05e`](https://github.com/kvcache-ai/sglang/commit/634b05e1878ef1eec5724cf8df9675443c00f84b) fix(packaging): preserve wheel-relative CUDA payload paths - [`d9749d3`](https://github.com/kvcache-ai/sglang/commit/d9749d35097f8e4aa23f2d28f2163431f8e886b7) style(kimi): format tokenizer factory call for CI - [`95eae11`](https://github.com/kvcache-ai/sglang/commit/95eae119ef7bd808310ed7bb3238434df84aa181) test: focus Kimi tokenizer and payload layout coverage ### 📊 Changes **4 files changed** (+102 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/constrained/llguidance_backend.py` (+20 -1) 📝 `sgl-kernel/python/sgl_kernel/payload_runtime.py` (+4 -2) ➕ `test/registered/unit/test_llguidance_kimi_tokenizer.py` (+50 -0) ➕ `test/registered/unit/test_payload_runtime_layout.py` (+28 -0) </details> ### 📄 Description ## What does this PR do? - Use llguidance's native tiktoken support for Kimi tokenizers; other models keep the existing dispatch. - Resolve lazy native binaries using their wheel-relative paths, preserving bundled library lookup. Legacy manifests remain supported. Two production files, +24/-3 lines, with two focused test files. The packaging counterpart is kvcache-ai/ktransformers#2204. ## Tests - Focused tokenizer/payload tests: 6 passed in a fresh wheel-only environment. - Original Kimi tokenizer: multilingual token IDs, EOS, vocabulary, and JSON grammar acceptance/rejection passed. - Fresh native wheels: dependency / architecture audit and eight concurrent cold imports passed. - Clean training/serving installation passed. Qwen two-GPU, Kimi four-GPU, and public FP8 GLM-5.3-Flash eight-GPU generation and JSON-schema checks passed. - Kimi loaded both ordinary and expert LoRA from a freshly resumed training checkpoint; all converted tensors passed exact comparison. Full Neko style validation remains pending. No model weights, attention kernels, or training runtime changes. Not a release sign-off. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:07:10 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#96

No description provided.

Rows
Columns