kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #117] 压测出现WC状态和QP错误 #58

New issue

Closed

opened 2026-10-29 21:08:33 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:08:33 +00:00

Owner

Originally created by @power-more on GitHub (Feb 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/117

双向读文件传数据压测
int RdmaContext::poll(int num_entries, ibv_wc *wc, int cq_index) {
int nr_poll = ibv_poll_cq(cq_list_[cq_index].native, num_entries, wc);
if (nr_poll < 0) {
LOG(ERROR) << "Failed to poll CQ " << cq_index << " of device "
<< device_name_;
return ERR_CONTEXT;
}
return nr_poll;
} 其中ibv_wc wc[kPollCount]; wc[i].status = [IBV_WC_GENERAL_ERR] = "general error",

以及
int num_events = epoll_wait(context_.eventFd(), &event, 1, 100);
if (event.data.fd == context_.context()->async_fd)
ibv_async_event event;
ibv_get_async_event(context_.context(), &event)
ibv_get_async_event后发现event_type是[IBV_EVENT_QP_FATAL] = "local work queue catastrophic error"，

Originally created by @power-more on GitHub (Feb 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/117 双向读文件传数据压测 int RdmaContext::poll(int num_entries, ibv_wc *wc, int cq_index) { int nr_poll = ibv_poll_cq(cq_list_[cq_index].native, num_entries, wc); if (nr_poll < 0) { LOG(ERROR) << "Failed to poll CQ " << cq_index << " of device " << device_name_; return ERR_CONTEXT; } return nr_poll; } 其中ibv_wc wc[kPollCount]; wc[i].status = [IBV_WC_GENERAL_ERR] = "general error", 以及 int num_events = epoll_wait(context_.eventFd(), &event, 1, 100); if (event.data.fd == context_.context()->async_fd) ibv_async_event event; ibv_get_async_event(context_.context(), &event) ibv_get_async_event后发现event_type是[IBV_EVENT_QP_FATAL] = "local work queue catastrophic error"，

bsakaguchi

2026-10-29 21:08:33 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@power-more commented on GitHub (Feb 22, 2025):

E0222 20:07:21.119704 2179433 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f5cfc000000, length: 65536, dest_addr: 140123979120640, local_nic: erdma_0, peer_nic: 10.0.0.41:7712@erdma_0, dest_rkey: 39908608, retry_cnt: 0): general error
W0222 20:07:21.119717 2179435 worker_pool.cpp:399] Worker: Received context async event local work queue catastrophic error for context erdma_0

@power-more commented on GitHub (Feb 22, 2025): E0222 20:07:21.119704 2179433 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f5cfc000000, length: 65536, dest_addr: 140123979120640, local_nic: erdma_0, peer_nic: 10.0.0.41:7712@erdma_0, dest_rkey: 39908608, retry_cnt: 0): general error W0222 20:07:21.119717 2179435 worker_pool.cpp:399] Worker: Received context async event local work queue catastrophic error for context erdma_0

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@alogfans commented on GitHub (Feb 23, 2025):

IBV_WC_GENERAL_ERR: This event is generated when there is a transport error which cannot be described by the other specific events discussed here.

你可以在其他的网络环境中（如 Mellanox 或普通 RoCEv2）尝试复现一下。根据提供的信息来看可能是driver方面的问题。

另外，依据此前类似问题的解决经验，可以尝试调节一些环境变量，特别是调高 MC_MAX_CQE_PER_CTX 的值，适度调低 MC_MAX_WR 的值，确认一下是否是因为 CQ 溢出修复不彻底所致。

@alogfans commented on GitHub (Feb 23, 2025): > IBV_WC_GENERAL_ERR: This event is generated when there is a transport error which cannot be described by the other specific events discussed here. 你可以在其他的网络环境中（如 Mellanox 或普通 RoCEv2）尝试复现一下。根据提供的信息来看可能是driver方面的问题。另外，依据此前类似问题的解决经验，可以尝试调节[一些环境变量](https://github.com/kvcache-ai/Mooncake/blob/main/doc/zh/transfer-engine.md#%E9%AB%98%E7%BA%A7%E8%BF%90%E8%A1%8C%E6%97%B6%E9%80%89%E9%A1%B9)，特别是调高 MC_MAX_CQE_PER_CTX 的值，适度调低 MC_MAX_WR 的值，确认一下是否是因为 CQ 溢出修复不彻底所致。

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@power-more commented on GitHub (Feb 24, 2025):

看起来不行。我重新设置了MC_MAX_CQE_PER_CTX:65535，MC_MAX_WR:1，已经是最悬殊的差距了，仍然复现。而且和MC_MAX_CQE_PER_CTX:2048，MC_MAX_WR:1024观察到的现象类似，前者也并没有多跑几秒。仅从测试效果来看，不一定是这个原因。
退一步讲，即使真的是填满 CQ cache的问题，那持续压测是否总会有跑满的时候呢？

@power-more commented on GitHub (Feb 24, 2025): 看起来不行。我重新设置了MC_MAX_CQE_PER_CTX:65535，MC_MAX_WR:1，已经是最悬殊的差距了，仍然复现。而且和MC_MAX_CQE_PER_CTX:2048，MC_MAX_WR:1024观察到的现象类似，前者也并没有多跑几秒。仅从测试效果来看，不一定是这个原因。退一步讲，即使真的是填满 CQ cache的问题，那持续压测是否总会有跑满的时候呢？

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@stmatengss commented on GitHub (Feb 26, 2025):

这是官方Spec里的一个解释，有一个Workaround的方式是重新建连，需要后续继续测试该问题。

@stmatengss commented on GitHub (Feb 26, 2025): ![Image](https://github.com/user-attachments/assets/b28816a5-8ac8-4697-a120-b5ecce5cbb7e) 这是官方Spec里的一个解释，有一个Workaround的方式是重新建连，需要后续继续测试该问题。

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@power-more commented on GitHub (Mar 27, 2025):

对于对端dereg再reg的mr，须先syncSegmentCache再submitTransfer

@power-more commented on GitHub (Mar 27, 2025): 对于对端dereg再reg的mr，须先syncSegmentCache再submitTransfer

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 29, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 29, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:11 +00:00

[PR #58] [MERGED] Bump golang.org/x/crypto to 0.31.0 & golang.org/x/net to v0.33.0 #1302