[GH-ISSUE #4454] [Bug]: Handshake daemon can kill the whole process with SIGPIPE when a peer resets mid-reply #1128

New issue

Closed

opened 2026-10-29 21:23:09 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:09 +00:00

Owner

Originally created by @he-yufeng on GitHub (Oct 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4454

The P2P handshake daemon answers every accepted frame with writeString(conn_fd, ...) and then half-closes (the listener loop in transfer_metadata_plugin.cpp). writeString goes through writeFully in common.h, a raw write() loop, and nothing in the transfer engine ignores SIGPIPE or sends with MSG_NOSIGNAL. If the peer resets the connection while the daemon is mid-reply (the client timed out and closed, crashed, or closed with reply bytes still unread), the daemon's next write() raises SIGPIPE and the whole process dies. In a vLLM-style deployment that process is the transfer engine host, so one flaky peer can take down a worker.

I hit this from the test side first: the flood test on #4453 closed each connection after a single bare read(), which can return after just the frame header. When that happened while the daemon was still writing the payload, the test process died with SIGPIPE on the CTest leg. The daemon path is the same writeString with untrusted peers on the RPC port, so the exposure is not test-only.

Minimal repro against a running P2PHANDSHAKE daemon: connect, send one valid Notify frame, close immediately without reading; repeat until the reply write lands after the RST (load-dependent).

Two small directions:

Switch the loop in writeFully to send(..., MSG_NOSIGNAL) (SO_NOSIGPIPE setsockopt on macOS, which has no MSG_NOSIGNAL). Surgical, and it also hardens the client side, which uses the same helper.
Ignore SIGPIPE once at daemon start (signal(SIGPIPE, SIG_IGN)), which covers every other socket write in the process too.

Happy to send a PR once there is a preferred direction.

Environment: current main, Linux x86_64, first observed on the ubuntu-22.04 CTest leg.

Originally created by @he-yufeng on GitHub (Oct 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4454 The P2P handshake daemon answers every accepted frame with `writeString(conn_fd, ...)` and then half-closes (the listener loop in transfer_metadata_plugin.cpp). `writeString` goes through `writeFully` in common.h, a raw `write()` loop, and nothing in the transfer engine ignores SIGPIPE or sends with MSG_NOSIGNAL. If the peer resets the connection while the daemon is mid-reply (the client timed out and closed, crashed, or closed with reply bytes still unread), the daemon's next `write()` raises SIGPIPE and the whole process dies. In a vLLM-style deployment that process is the transfer engine host, so one flaky peer can take down a worker. I hit this from the test side first: the flood test on #4453 closed each connection after a single bare `read()`, which can return after just the frame header. When that happened while the daemon was still writing the payload, the test process died with SIGPIPE on the CTest leg. The daemon path is the same `writeString` with untrusted peers on the RPC port, so the exposure is not test-only. Minimal repro against a running P2PHANDSHAKE daemon: connect, send one valid Notify frame, close immediately without reading; repeat until the reply write lands after the RST (load-dependent). Two small directions: 1. Switch the loop in `writeFully` to `send(..., MSG_NOSIGNAL)` (SO_NOSIGPIPE setsockopt on macOS, which has no MSG_NOSIGNAL). Surgical, and it also hardens the client side, which uses the same helper. 2. Ignore SIGPIPE once at daemon start (`signal(SIGPIPE, SIG_IGN)`), which covers every other socket write in the process too. Happy to send a PR once there is a preferred direction. Environment: current main, Linux x86_64, first observed on the ubuntu-22.04 CTest leg.

bsakaguchi

2026-10-29 21:23:09 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 2, 2026):

Thanks for opening this issue, @he-yufeng!

Field	Value
Issue	#4454
GitHub user ID	`40085740`
Reporter	@he-yufeng

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 2, 2026): Thanks for opening this issue, @he-yufeng! | Field | Value | |-------|-------| | **Issue** | #4454 | | **GitHub user ID** | `40085740` | | **Reporter** | @he-yufeng | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:13 +00:00

[PR #1128] [CLOSED] [TE] fix(rdma): pre-compute slice_count to avoid race condition #2028