kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1381] unit tests #926

New issue

Open

opened 2026-10-29 14:46:13 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:46:13 +00:00

Owner

Originally created by @createthis on GitHub (Jun 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1381

Originally assigned to: @Azure-Tang on GitHub.

I've noticed commit 8dc1ab9 (Apr 14th) is WAY more reliable on my hardware with DeepSeek-V3-0324:671b-q4_k_m than current 64ec0ec (June 11th). I can get 50k to 80k long context from 8dc1ab9. However, I'm lucky to get 24k to 29k long context from 64ec0ec.

My hardware is dual EPYC 9355 CPUs with 768gb RAM and a 24gb 3090.

I've also noticed that I can easily write code that runs, but introduces a lot of instability and causes models to crash. For instance, I wrote code that backported R1-0528 support all the way to the March 15th commit bda9cf1 - it ran, but it was useless because it went insane after just a turn or two.

How can we/I write unit tests that will help us verify that we aren't introducing problems when we make changes?

Originally created by @createthis on GitHub (Jun 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1381 Originally assigned to: @Azure-Tang on GitHub. I've noticed commit [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9) (Apr 14th) is WAY more reliable on my hardware with `DeepSeek-V3-0324:671b-q4_k_m` than current [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7) (June 11th). I can get 50k to 80k long context from [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9). However, I'm lucky to get 24k to 29k long context from [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7). My hardware is dual EPYC 9355 CPUs with 768gb RAM and a 24gb 3090. I've also noticed that I can easily write code that runs, but introduces a lot of instability and causes models to crash. For instance, I wrote code that backported R1-0528 support all the way to the March 15th commit `bda9cf1` - it ran, but it was useless because it went insane after just a turn or two. How can we/I write unit tests that will help us verify that we aren't introducing problems when we make changes?

bsakaguchi commented

2026-10-29 14:46:14 +00:00

Author

Owner

@ErvinXie commented on GitHub (Jun 12, 2025):

We have some unit tests, in examples, to test cpu moe correctness.

@ErvinXie commented on GitHub (Jun 12, 2025): We have some unit tests, in [`examples`](https://github.com/kvcache-ai/ktransformers/tree/main/csrc/ktransformers_ext/examples), to test cpu moe correctness.

bsakaguchi commented

2026-10-29 14:46:14 +00:00

Author

Owner

@createthis commented on GitHub (Jun 12, 2025):

Hmm. Seems broken in 8dc1ab9.

(ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py 
Traceback (most recent call last):
  File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 101, in <module>
    moe.forward( 
    ^^^^^^^^^^^^
TypeError: forward(): incompatible function arguments. The following argument types are supported:
    1. (self: cpuinfer_ext.moe.MOE, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int, arg5: int, arg6: int) -> tuple[int, int]

Invoked with: <cpuinfer_ext.moe.MOE object at 0x78550c869ab0>, 30, 6, 1039414144, 1039409344, 1040360832, 1040053568

@createthis commented on GitHub (Jun 12, 2025): Hmm. Seems broken in `8dc1ab9`. ``` (ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py Traceback (most recent call last): File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 101, in <module> moe.forward( ^^^^^^^^^^^^ TypeError: forward(): incompatible function arguments. The following argument types are supported: 1. (self: cpuinfer_ext.moe.MOE, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int, arg5: int, arg6: int) -> tuple[int, int] Invoked with: <cpuinfer_ext.moe.MOE object at 0x78550c869ab0>, 30, 6, 1039414144, 1039409344, 1040360832, 1040053568 ```

bsakaguchi commented

2026-10-29 14:46:14 +00:00

Author

Owner

@createthis commented on GitHub (Jun 12, 2025):

I had a moment during sprint planning to debug this. I made this change:

diff --git a/csrc/ktransformers_ext/examples/test_moe.py b/csrc/ktransformers_ext/examples/test_moe.py
index 3fa4dbd..f05b078 100644
--- a/csrc/ktransformers_ext/examples/test_moe.py
+++ b/csrc/ktransformers_ext/examples/test_moe.py
@@ -97,13 +97,16 @@ with torch.inference_mode(mode=True):
         input = input / 100
         
         moe = moes[i % layer_num]
+        ws = getattr(moe, "workspace_ptr", lambda: 0)()
+
         CPUInfer.submit(
             moe.forward( 
                 qlen,
                 n_routed_experts, 
                 expert_ids.data_ptr(), 
                 weights.data_ptr(), 
-                input.data_ptr(), 
+                input.data_ptr(),
+                ws,
                 output.data_ptr()
             )
         )

And got this assertion failure:

(ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py 
diff =  tensor(1., dtype=torch.float16)
Traceback (most recent call last):
  File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 124, in <module>
    assert(diff < 0.001)
           ^^^^^^^^^^^^
AssertionError
(ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$

No clue if that's a legitimate bug being caught or if the test is out of date. Again, this is with commit 8dc1ab9. I'll try 64ec0ec next since I'm stuck in planning.

@createthis commented on GitHub (Jun 12, 2025): I had a moment during sprint planning to debug this. I made this change: ```diff diff --git a/csrc/ktransformers_ext/examples/test_moe.py b/csrc/ktransformers_ext/examples/test_moe.py index 3fa4dbd..f05b078 100644 --- a/csrc/ktransformers_ext/examples/test_moe.py +++ b/csrc/ktransformers_ext/examples/test_moe.py @@ -97,13 +97,16 @@ with torch.inference_mode(mode=True): input = input / 100 moe = moes[i % layer_num] + ws = getattr(moe, "workspace_ptr", lambda: 0)() + CPUInfer.submit( moe.forward( qlen, n_routed_experts, expert_ids.data_ptr(), weights.data_ptr(), - input.data_ptr(), + input.data_ptr(), + ws, output.data_ptr() ) ) ``` And got this assertion failure: ```bash (ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py diff = tensor(1., dtype=torch.float16) Traceback (most recent call last): File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 124, in <module> assert(diff < 0.001) ^^^^^^^^^^^^ AssertionError (ktransformers_mc26) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ ``` No clue if that's a legitimate bug being caught or if the test is out of date. Again, this is with commit `8dc1ab9`. I'll try `64ec0ec` next since I'm stuck in planning.

bsakaguchi commented

2026-10-29 14:46:15 +00:00

Author

Owner

@createthis commented on GitHub (Jun 12, 2025):

Same result with the same patch at commit 64ec0ec:

(ktransformers_mc26_head) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py 
diff =  tensor(1., dtype=torch.float16)
Traceback (most recent call last):
  File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 124, in <module>
    assert(diff < 0.001)
           ^^^^^^^^^^^^
AssertionError
(ktransformers_mc26_head) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$

@createthis commented on GitHub (Jun 12, 2025): Same result with the same patch at commit `64ec0ec`: ```bash (ktransformers_mc26_head) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ python3 test_moe.py diff = tensor(1., dtype=torch.float16) Traceback (most recent call last): File "/home/jesse/ktransformers/ktransformers/csrc/ktransformers_ext/examples/test_moe.py", line 124, in <module> assert(diff < 0.001) ^^^^^^^^^^^^ AssertionError (ktransformers_mc26_head) jesse@larry:~/ktransformers/ktransformers/csrc/ktransformers_ext/examples$ ```

bsakaguchi commented

2026-10-29 14:46:15 +00:00

Author

Owner

@createthis commented on GitHub (Jun 12, 2025):

Time for CI/CD?

@createthis commented on GitHub (Jun 12, 2025): Time for CI/CD?

bsakaguchi referenced this issue

2026-10-29 14:53:09 +00:00

[PR #926] [MERGED] Update gate.py #1518