kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1822] [Bug]: EFA Segmentation Fault on Loopback Test #522

New issue

Closed

opened 2026-10-29 21:15:50 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:15:50 +00:00

Owner

Originally created by @leoleoasd on GitHub (Apr 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1822

Bug Report

I'm encountering seg fault when trying to run loopback write test. This is what claude code gave me from analysing the code dump:

EFA Transport LoopbackWrite segfaults in fi_cq_read due to race with fi_enable
Environment

AWS p5en.48xlarge (16 EFA devices, rdmap* naming)
Docker container with EFA installer 1.38.0
libfabric 1.22.0amzn4.0 (/opt/amazon/efa/lib/libfabric.so.1)
Ubuntu 24.04, CUDA 12.9, GCC 13.3
Mooncake built with -DUSE_EFA=ON -DUSE_CUDA=ON
Bug reproduces on every commit that includes EFA support (from the initial 4136d2b through v0.3.10.post1)
Reproduction
./build/mooncake-transfer-engine/tests/efa_transport_test
The InstallTransport test passes. The LoopbackWrite test segfaults every time.

Root Cause (from core dump)
There is a race condition between two threads:
Thread 1 (CQ poller worker, thread_id=8) — crashes here:
#0  ?? () from libfabric.so.1
#4  fi_cq_read(cq=0x56060ff0dd20, count=64) at fi_eq.h:396
#5  EfaContext::pollCq(max_entries=64, cq_index=0) at efa_context.cpp:503
#6  EfaTransport::workerThreadFunc(thread_id=8) at efa_transport.cpp:89
Thread 20 (main test thread) — concurrently creating an endpoint for the loopback connection:
#10 fi_enable(ep) at fi_endpoint.h:226
#11 EfaEndPoint::construct() at efa_endpoint.cpp:95
#12 EfaContext::endpoint(peer_nic_path="127.0.0.1:16168@rdmap88s0") at efa_context.cpp:371
#13 EfaContext::submitPostSend() at efa_context.cpp:469
#14 EfaTransport::submitTransferTask() at efa_transport.cpp:555
#18 EFATransportTest::submitAndWait(WRITE, length=4096) at efa_transport_test.cpp:124
#19 EFATransportTest_LoopbackWrite_Test::TestBody() at efa_transport_test.cpp:188
The CQ polling worker threads are started in EfaTransport::installTransport() (efa_transport.cpp:59) before the first endpoint is constructed. When the LoopbackWrite test triggers submitPostSend, it lazily creates a new EfaEndPoint and calls fi_enable() on it. Meanwhile, the CQ poller is already calling fi_cq_read() on the shared CQ. The fi_enable() call internally creates a QP via efadv_create_qp_ex() → ibv_cmd_create_qp_ex(), which modifies the CQ's internal state while fi_cq_read() is reading it, causing the segfault.

I'll be happy to share the core dump file if needed (it's 1.3GB)

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @leoleoasd on GitHub (Apr 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1822 ### Bug Report I'm encountering seg fault when trying to run loopback write test. This is what claude code gave me from analysing the code dump: EFA Transport LoopbackWrite segfaults in fi_cq_read due to race with fi_enable Environment - AWS p5en.48xlarge (16 EFA devices, rdmap* naming) - Docker container with EFA installer 1.38.0 - libfabric 1.22.0amzn4.0 (/opt/amazon/efa/lib/libfabric.so.1) - Ubuntu 24.04, CUDA 12.9, GCC 13.3 - Mooncake built with -DUSE_EFA=ON -DUSE_CUDA=ON - Bug reproduces on every commit that includes EFA support (from the initial 4136d2b through v0.3.10.post1) Reproduction ./build/mooncake-transfer-engine/tests/efa_transport_test The InstallTransport test passes. The LoopbackWrite test segfaults every time. ``` Root Cause (from core dump) There is a race condition between two threads: Thread 1 (CQ poller worker, thread_id=8) — crashes here: #0 ?? () from libfabric.so.1 #4 fi_cq_read(cq=0x56060ff0dd20, count=64) at fi_eq.h:396 #5 EfaContext::pollCq(max_entries=64, cq_index=0) at efa_context.cpp:503 #6 EfaTransport::workerThreadFunc(thread_id=8) at efa_transport.cpp:89 Thread 20 (main test thread) — concurrently creating an endpoint for the loopback connection: #10 fi_enable(ep) at fi_endpoint.h:226 #11 EfaEndPoint::construct() at efa_endpoint.cpp:95 #12 EfaContext::endpoint(peer_nic_path="127.0.0.1:16168@rdmap88s0") at efa_context.cpp:371 #13 EfaContext::submitPostSend() at efa_context.cpp:469 #14 EfaTransport::submitTransferTask() at efa_transport.cpp:555 #18 EFATransportTest::submitAndWait(WRITE, length=4096) at efa_transport_test.cpp:124 #19 EFATransportTest_LoopbackWrite_Test::TestBody() at efa_transport_test.cpp:188 The CQ polling worker threads are started in EfaTransport::installTransport() (efa_transport.cpp:59) before the first endpoint is constructed. When the LoopbackWrite test triggers submitPostSend, it lazily creates a new EfaEndPoint and calls fi_enable() on it. Meanwhile, the CQ poller is already calling fi_cq_read() on the shared CQ. The fi_enable() call internally creates a QP via efadv_create_qp_ex() → ibv_cmd_create_qp_ex(), which modifies the CQ's internal state while fi_cq_read() is reading it, causing the segfault. ``` I'll be happy to share the core dump file if needed (it's 1.3GB) ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:15:50 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:15:51 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 5, 2026):

Hi, @whn09, we lack experience in EFA. Could you help solve this issue?

@stmatengss commented on GitHub (Apr 5, 2026): Hi, @whn09, we lack experience in EFA. Could you help solve this issue?

bsakaguchi commented

2026-10-29 21:15:51 +00:00

Author

Owner

@whn09 commented on GitHub (Apr 7, 2026):

You can try this repo: https://github.com/whn09/kimi-k2-sglang

@whn09 commented on GitHub (Apr 7, 2026): You can try this repo: https://github.com/whn09/kimi-k2-sglang

bsakaguchi commented

2026-10-29 21:15:51 +00:00

Author

Owner

@whn09 commented on GitHub (Apr 7, 2026):

This is an EFA driver mismatch issue, and you can use this command to run your docker:

docker run --gpus all --device=/dev/infiniband --net=host --privileged -it -v /opt/amazon/efa:/opt/amazon/efa -e LD_LIBRARY_PATH=/opt/amazon/efa/lib:$LD_LIBRARY_PATH -v /lib/x86_64-linux-gnu/libefa.so.1:/lib/x86_64-linux-gnu/libefa.so.1 -v /lib/x86_64-linux-gnu/libefa.so:/lib/x86_64-linux-gnu/libefa.so -v /lib/x86_64-linux-gnu/libibverbs.so.1:/lib/x86_64-linux-gnu/libibverbs.so.1 YOUR_DOCKER_IMAGE /bin/bash

@whn09 commented on GitHub (Apr 7, 2026): This is an EFA driver mismatch issue, and you can use this command to run your docker: `docker run --gpus all --device=/dev/infiniband --net=host --privileged -it -v /opt/amazon/efa:/opt/amazon/efa -e LD_LIBRARY_PATH=/opt/amazon/efa/lib:$LD_LIBRARY_PATH -v /lib/x86_64-linux-gnu/libefa.so.1:/lib/x86_64-linux-gnu/libefa.so.1 -v /lib/x86_64-linux-gnu/libefa.so:/lib/x86_64-linux-gnu/libefa.so -v /lib/x86_64-linux-gnu/libibverbs.so.1:/lib/x86_64-linux-gnu/libibverbs.so.1 YOUR_DOCKER_IMAGE /bin/bash`

bsakaguchi commented

2026-10-29 21:15:51 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 7, 2026):

You can try this repo: https://github.com/whn09/kimi-k2-sglang

Thanks. I will try it

@stmatengss commented on GitHub (Apr 7, 2026): > You can try this repo: https://github.com/whn09/kimi-k2-sglang Thanks. I will try it

bsakaguchi commented

2026-10-29 21:15:51 +00:00

Author

Owner

@leoleoasd commented on GitHub (Apr 18, 2026):

Upgrading to the latest EFA driver helps. Thanks!

@leoleoasd commented on GitHub (Apr 18, 2026): Upgrading to the latest EFA driver helps. Thanks!