[GH-ISSUE #797] [Bug]: 💩💩💩💩💩💩 close blocked without any logging

bsakaguchi commented

2026-10-29 21:11:27 +00:00

Owner

Originally created by @ActivePeter on GitHub (Sep 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/797

Originally assigned to: @xiaguan on GitHub.

Bug Report

💩💩💩💩💩💩 There's always some unexpected error. We found many ways to check each of the error (not provided by Mooncake but supposed to be), but we still can't handle it ourselves because we can't properly close the old one.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @ActivePeter on GitHub (Sep 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/797 Originally assigned to: @xiaguan on GitHub. ### Bug Report 💩💩💩💩💩💩 There's always some unexpected error. We found many ways to check each of the error (not provided by Mooncake but supposed to be), but we still can't handle it ourselves because we can't properly close the old one. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:11:27 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:11:28 +00:00

Author

Owner

@mumupika commented on GitHub (Sep 2, 2025):

We have consistently encountered with the following problems which severely harms the stability of the system:

RDMA failed. In worker_pool.cpp RDMA encountered with work queue local access violation. We noticed that you provided a Fault Handling mechanism that provides the solution to the signal NIC problem under multi-NIC environment. Can you provide a more specific solutions or suggestions on configurations, i.e., the ways to configure the substitutional RDMA NIC?

After the transfer failed, we have provided a temporary solution that we will try to close the store that has setup and renew a new store (re-setup a store) to replace the old one. But under the condition of NIC malfunction, the close operation seems blocked and never send a return code to us. And the exceptions were raised by NCCL timeout. Maybe a timeout solution in DistributedStore.close() can be provided?

We are eager for your suggestions and sincerely appreciated your patience!

@mumupika commented on GitHub (Sep 2, 2025): We have consistently encountered with the following problems which severely harms the stability of the system: 1. RDMA failed. In `worker_pool.cpp` RDMA encountered with work queue local access violation. We noticed that you provided a **Fault Handling** mechanism that provides the solution to the signal NIC problem under multi-NIC environment. Can you provide a more specific solutions or suggestions on configurations, i.e., the ways to configure the substitutional RDMA NIC? <img width="741" height="48" alt="Image" src="https://github.com/user-attachments/assets/24cb0703-6147-4163-be4c-d167ec3ecd8d" /> 2. After the transfer failed, we have provided a temporary solution that we will try to close the store that has setup and renew a new store (re-setup a store) to replace the old one. But under the condition of NIC malfunction, the `close` operation seems blocked and never send a return code to us. And the exceptions were raised by NCCL timeout. Maybe a timeout solution in `DistributedStore.close()` can be provided? <img width="672" height="116" alt="Image" src="https://github.com/user-attachments/assets/7a7f1906-ef73-41a3-b40f-b2bd4f34e868" /> We are eager for your suggestions and sincerely appreciated your patience!

bsakaguchi commented

2026-10-29 21:11:28 +00:00

Author

Owner

@mumupika commented on GitHub (Sep 2, 2025):

It appears to me that this local access violation work queue error occurs from IBV_EVENT_QP_ACCESS_ERR. Suggested from NVIDIA's events documentation, this error may caused by:

Misaligned atomic request
Too many RDMA Read or Atomic requests R_Key violation
Length errors without immediate data

Which may possibly occurred from the user code. I am eagerly waiting for your suggestions! Thanks for your contribution and patience!

@mumupika commented on GitHub (Sep 2, 2025): It appears to me that this `local access violation work queue error` occurs from `IBV_EVENT_QP_ACCESS_ERR`. Suggested from [NVIDIA's events documentation](https://docs.nvidia.com/networking/display/rdmaawareprogrammingv17/events#src-34256581_Events-IBV_EVENT_QP_ACCESS_ERR), this error may caused by: - Misaligned atomic request - Too many RDMA Read or Atomic requests R_Key violation - Length errors without immediate data Which may possibly occurred from the user code. I am eagerly waiting for your suggestions! Thanks for your contribution and patience! <img width="803" height="349" alt="Image" src="https://github.com/user-attachments/assets/064315d4-7655-414c-977e-d49c21e7e114" />

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@alogfans commented on GitHub (Sep 3, 2025):

In our previous observations, the local access violation work queue error may have been caused by either (1) some network cards are unable to use, you can use --disaggregation-ib-device to set a whitelist (https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/sglang-integration-v1.md); or (2) some inference processes crashed during execution.

@alogfans commented on GitHub (Sep 3, 2025): In our previous observations, the `local access violation work queue error` may have been caused by either (1) some network cards are unable to use, you can use `--disaggregation-ib-device` to set a whitelist (https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/sglang-integration-v1.md); or (2) some inference processes crashed during execution.

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@mumupika commented on GitHub (Sep 3, 2025):

@alogfans Hello, Thanks for your patient and advice above!

We have done several reproduces that after transfer failed by local access violation work queue error, we use a seperate thread to execute DistributedStore.close(), in order to avoid from the crash. But the process missed that... It seems that was blocked by the cpp backend and when torch send signal 15 SIGTERM to this process, the store_py.cpp handled this signal and clean the resources. Did your team encountered with this problem before or this is also a NETWORK PROBLEM? Thanks for your patients and suggestions!

And I am apology for the logs below... The machine is not connected to internet so I have to photo it...

PID 3555053 alives here

We start the closing thread like this after we encountered local access violation work queue error...

def close_nonblock():
    def close_store():
        try:
            retcode = self._store.close()
            if retcode == 0:
                # success
            else:
                # fail
    
    close_thread = threading.Thread(close_store, daemon = True)
    close_thread.start()
    close_thread.join(timeout = timeout)
    # ...

nonblocking calls here since we use a thread in the background and wait for limited time for the thread to join
(Apologies for watermarks that only pieces of log were typed from picture QAQ)

[2025-09-03 00:53:12,503] KVCache WARNING:
====== Mooncake store try_get trial =====
With error:
ValueSizeChangedError(4005: Value size changed for key '/mpscchan_3_producer_1_msg_301' from 1447374 to 0 ....
With key: /mpscchan_3_producer_1_msg_301
Retry with new_store...(mooncake.py:273:py_kvcache)(retry_operation:/nvfile-heatstor...
2025-09-03 00:53:12,5041 KVCache DEBUG:  closing the store noblocking... (mooncake.py

And I cannot see the PID alive log in our python interface after that until SIGTERM is sent

Have your teams ever encountered with DistributedStore.close() blocked the output log on the above python process? Is there any suggestions available? Sincerely thanks for your advice!

@mumupika commented on GitHub (Sep 3, 2025): @alogfans Hello, Thanks for your patient and advice above! We have done several reproduces that after transfer failed by `local access violation work queue error`, we use a seperate thread to execute `DistributedStore.close()`, in order to avoid from the crash. But the process missed that... It seems that was blocked by the cpp backend and when torch send signal 15 SIGTERM to this process, the `store_py.cpp` handled this signal and clean the resources. Did your team encountered with this problem before or this is also a **NETWORK PROBLEM**? Thanks for your patients and suggestions! And I am apology for the logs below... The machine is not connected to internet so I have to photo it... PID 3555053 alives here <img width="275" height="147" alt="Image" src="https://github.com/user-attachments/assets/be3c0bdd-6337-435c-a21a-26295976e898" /> We start the closing thread like this after we encountered `local access violation work queue error`... ```python def close_nonblock(): def close_store(): try: retcode = self._store.close() if retcode == 0: # success else: # fail close_thread = threading.Thread(close_store, daemon = True) close_thread.start() close_thread.join(timeout = timeout) # ... ``` nonblocking calls here since we use a thread in the background and wait for limited time for the thread to join (Apologies for watermarks that only pieces of log were typed from picture QAQ) ``` [2025-09-03 00:53:12,503] KVCache WARNING: ====== Mooncake store try_get trial ===== With error: ValueSizeChangedError(4005: Value size changed for key '/mpscchan_3_producer_1_msg_301' from 1447374 to 0 .... With key: /mpscchan_3_producer_1_msg_301 Retry with new_store...(mooncake.py:273:py_kvcache)(retry_operation:/nvfile-heatstor... 2025-09-03 00:53:12,5041 KVCache DEBUG: closing the store noblocking... (mooncake.py ``` And I cannot see the PID alive log in our python interface after that until SIGTERM is sent <img width="813" height="195" alt="Image" src="https://github.com/user-attachments/assets/1e1aa6fe-6545-4466-a325-cc5e47ae7414" /> Have your teams ever encountered with `DistributedStore.close()` blocked the output log on the above python process? Is there any suggestions available? Sincerely thanks for your advice!

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@alogfans commented on GitHub (Sep 3, 2025):

@mumupika I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can @ykwd respond on it?

Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals?

@alogfans commented on GitHub (Sep 3, 2025): @mumupika I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can @ykwd respond on it? Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals?

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@mumupika commented on GitHub (Sep 3, 2025):

@mumupika I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can @ykwd respond on it?

Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals?

Thanks! But it seems that the SIGTERM was sent by NCCL when the timeout reaches and the programs above is out of my control QAQ (which is managed by other teams). I will update as soon as I received new logs and training results from others.

Sincerely appreciate your patience and reply! @alogfans

@mumupika commented on GitHub (Sep 3, 2025): > [@mumupika](https://github.com/mumupika) I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can [@ykwd](https://github.com/ykwd) respond on it? > > Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals? Thanks! But it seems that the SIGTERM was sent by NCCL when the timeout reaches and the programs above is out of my control QAQ (which is managed by other teams). I will update as soon as I received new logs and training results from others. Sincerely appreciate your patience and reply! @alogfans

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 3, 2025):

@mumupika I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can @ykwd respond on it?

Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals?

Me neither. This is strange.

@ykwd commented on GitHub (Sep 3, 2025): > [@mumupika](https://github.com/mumupika) I didn't encounter such error before. In the side of Mooncake Transfer Engine, a close operation only destroy all rdma resources, which should not take too much time. In the Mooncake Store side, can [@ykwd](https://github.com/ykwd) respond on it? > > Failure to shutdown pervious instance may be lead by hooking the SIGTERM signal (but I didn't use it in Transfer Engine). Can you try other signals? Me neither. This is strange.

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@xiaguan commented on GitHub (Sep 15, 2025):

https://github.com/kvcache-ai/Mooncake/pull/840, you could try this

@xiaguan commented on GitHub (Sep 15, 2025): https://github.com/kvcache-ai/Mooncake/pull/840, you could try this

bsakaguchi commented

2026-10-29 21:11:29 +00:00

Author

Owner

@xiaguan commented on GitHub (Sep 17, 2025):