[GH-ISSUE #1400] [Bug]: check_status KeyError #450

New issue

Closed

opened 2026-10-29 21:14:40 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:14:40 +00:00

Owner

Originally created by @peleg-yair on GitHub (Jan 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1400

Originally assigned to: @ShangmingCai on GitHub.

Bug Report

Version: mooncake-transfer-engine 0.3.7.post2
Running nvidia/DeepSeek-V3-0324-NVFP4 with SGLang P/D and got this KeyError on prefill workers:

[2026-01-16 22:03:02 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2682, in run_scheduler_process
    scheduler.event_loop_overlap_disagg_prefill()
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/prefill.py", line 378, in event_loop_overlap_disagg_prefill
    self.process_disagg_prefill_inflight_queue()
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/prefill.py", line 518, in process_disagg_prefill_inflight_queue
    polls = poll_and_all_reduce(
            ^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/utils.py", line 49, in poll_and_all_reduce
    polls = [int(poller.poll()) for poller in pollers]
                 ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1120, in poll
    status = self.kv_mgr.check_status(self.bootstrap_room)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1019, in check_status
    return self.request_status[bootstrap_room]
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 5497793356111655473

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @peleg-yair on GitHub (Jan 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1400 Originally assigned to: @ShangmingCai on GitHub. ### Bug Report Version: `mooncake-transfer-engine 0.3.7.post2` Running `nvidia/DeepSeek-V3-0324-NVFP4` with SGLang P/D and got this KeyError on `prefill` workers: ``` [2026-01-16 22:03:02 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2682, in run_scheduler_process scheduler.event_loop_overlap_disagg_prefill() File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/prefill.py", line 378, in event_loop_overlap_disagg_prefill self.process_disagg_prefill_inflight_queue() File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/prefill.py", line 518, in process_disagg_prefill_inflight_queue polls = poll_and_all_reduce( ^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/utils.py", line 49, in poll_and_all_reduce polls = [int(poller.poll()) for poller in pollers] ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1120, in poll status = self.kv_mgr.check_status(self.bootstrap_room) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1019, in check_status return self.request_status[bootstrap_room] ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ KeyError: 5497793356111655473 ``` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:14:40 +00:00

closed this issue
added the
stale

auto-closed

bug
labels

bsakaguchi commented

2026-10-29 21:14:41 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jan 19, 2026):

@peleg-yair Which SGlang version? If the version is new, maybe check whether your router produces two requests with the identical bootstrap room: 5497793356111655473?

@ShangmingCai commented on GitHub (Jan 19, 2026): @peleg-yair Which SGlang version? If the version is new, maybe check whether your router produces two requests with the identical bootstrap room: 5497793356111655473?

bsakaguchi commented

2026-10-29 21:14:41 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 9, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 9, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:14:41 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 16, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jun 16, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.