[PR #1405] [MERGED] add troubleshooting of exceeding ulimit

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#2225

bsakaguchi commented

2026-10-29 21:33:49 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1405
Author: @zhangzuo21
Created: 1/19/2026
Status: ✅ Merged
Merged: 1/20/2026
Merged by: @stmatengss

Base: main ← Head: exceed-ulimit

📝 Commits (2)

f4ac2fe add troubleshooting of exceeding ulimit
c7f77c9 Update docs/source/troubleshooting/troubleshooting.md

📊 Changes

1 file changed (+18 additions, -1 deletions)

View changed files

📝 docs/source/troubleshooting/troubleshooting.md (+18 -1)

📄 Description

Description

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- [✅ ] Documentation update
- Other

How Has This Been Tested?

Summary

Add troubleshooting guide for ulimit-related memory allocation failures in RDMA resource initialization.

Problem

When initializing Mooncake Store with large memory regions, users may encounter memory allocation failures even when the physical memory is available. This occurs when the total memory requirement (number of registered RDMA devices × requested space per device) exceeds the system's ulimit restrictions, particularly the max locked memory limit.

This issue can be confusing because:

The error messages about "failed to allocate space" don't clearly indicate ulimit as the root cause
Users may have sufficient physical memory but still hit these limits
The problem becomes more severe when multiple RDMA devices are registered

Changes

Added a new troubleshooting entry (point 5) in the "RDMA Resource Initialization" section
Provides clear diagnostic steps to identify ulimit-related issues
Documents multiple solution approaches:
- Running with elevated privileges (root)
- Temporarily modifying ulimit settings
- Launching multiple store instances with smaller allocations
- Permanently configuring limits via /etc/security/limits.conf

Impact

This documentation improvement will help users:

Quickly diagnose ulimit-related memory allocation failures
Choose appropriate solutions based on their deployment environment
Avoid trial-and-error when configuring large memory regions

Testing

Verified the documentation follows the existing format and style
Confirmed the technical accuracy of diagnostic commands and solutions

Checklist

I have performed a self-review of my own code.
I have formatted my own code using ./scripts/code_format.sh before submitting.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1405 **Author:** [@zhangzuo21](https://github.com/zhangzuo21) **Created:** 1/19/2026 **Status:** ✅ Merged **Merged:** 1/20/2026 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `exceed-ulimit` --- ### 📝 Commits (2) - [`f4ac2fe`](https://github.com/kvcache-ai/Mooncake/commit/f4ac2fe6756c288e99a1659e147f3973d8f5c636) add troubleshooting of exceeding ulimit - [`c7f77c9`](https://github.com/kvcache-ai/Mooncake/commit/c7f77c987aba85b0cb13e27f2ae635cfbada504f) Update docs/source/troubleshooting/troubleshooting.md ### 📊 Changes **1 file changed** (+18 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `docs/source/troubleshooting/troubleshooting.md` (+18 -1) </details> ### 📄 Description ## Description  ## Type of Change * Types - [ ] Bug fix - [ ] New feature - [ ] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [✅ ] Documentation update - [ ] Other ## How Has This Been Tested? ## Summary Add troubleshooting guide for ulimit-related memory allocation failures in RDMA resource initialization. ## Problem When initializing Mooncake Store with large memory regions, users may encounter memory allocation failures even when the physical memory is available. This occurs when the total memory requirement (number of registered RDMA devices × requested space per device) exceeds the system's ulimit restrictions, particularly the `max locked memory` limit. This issue can be confusing because: - The error messages about "failed to allocate space" don't clearly indicate ulimit as the root cause - Users may have sufficient physical memory but still hit these limits - The problem becomes more severe when multiple RDMA devices are registered ## Changes - Added a new troubleshooting entry (point 5) in the "RDMA Resource Initialization" section - Provides clear diagnostic steps to identify ulimit-related issues - Documents multiple solution approaches: - Running with elevated privileges (root) - Temporarily modifying ulimit settings - Launching multiple store instances with smaller allocations - Permanently configuring limits via `/etc/security/limits.conf` ## Impact This documentation improvement will help users: - Quickly diagnose ulimit-related memory allocation failures - Choose appropriate solutions based on their deployment environment - Avoid trial-and-error when configuring large memory regions ## Testing - Verified the documentation follows the existing format and style - Confirmed the technical accuracy of diagnostic commands and solutions ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have formatted my own code using `./scripts/code_format.sh` before submitting. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

closed this issue
added the
pull-request
label

Rows
Columns

[PR #1405] [MERGED] add troubleshooting of exceeding ulimit #2225