resume cycle #57

New issue

Closed

opened 2026-10-29 14:18:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:56 +00:00

Owner

Originally created by @YXalix on GitHub (Aug 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/171

Affected component

ublk / ublk-daemon

AgentENV version

Version or commit:0475f403b119c29d3b74aa32b5c10dff07c68493
Local modifications: none

Host environment

OS:debian(velinux)
Kernel (uname -a):6.6.95
Architecture:ARM64
Bare metal, VM, or cloud:Bare metal
ublk support:ublk_drv loaded
CPU:kunpeng 950
Memory: 1.5T
Storage/filesystem:ext4

Operation and configuration

During stress testing involving 600 virtual machines continuously executing pause and resume cycles, the uvm-ublk-daemon process exhibits rapid and uncontrolled memory growth in anon_mmap at a rate of 20–40 MB/s, eventually reaching tens of gigabytes.

Memory allocations captured using memleak-bpfcc indicate that the majority of unreleased memory stems from overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write and tokio::fs::read_dir.

Steps to reproduce

Sequentially launch 600 virtual machines:
- Keep 100 VMs in the running state.
- Keep the remaining 500 VMs in the pause state.
Run a continuous loop of pause + resume operations across all 600 VMs.
Observe the memory consumption metrics of uvm-ublk-daemon.

Expected behavior

uvm-ublk-daemon memory consumption should remain stable within a bounded range during repeated VM pause/resume cycles.

Actual behavior

anon_mmap grows linearly at 20–40 MB/s.

Logs and diagnostics

aenv/logs/ublk-daemon.log


2026-08-14T10:02:06.598578Z  INFO uvm_ublk_daemon::server: ReleaseOverlaybd { dev_id: 224 }
2026-08-14T10:02:06.598602Z  INFO uvm_ublk_daemon::server: releasing shared device (refcount reached 0) dev_id=224
2026-08-14T10:02:06.598625Z DEBUG uvm_ublk_daemon::server: cleared page cache path=/dev/ublkb224
2026-08-14T10:02:06.599138Z  INFO uvm_ublk_daemon::server: idle overlaybd pool is full; stopping returned device dev_id=224 high_watermark=128
2026-08-14T10:02:06.599144Z  INFO uvm_ublk_daemon::server: stopping pooled overlaybd device during shutdown dev_id=224
2026-08-14T10:02:06.599149Z DEBUG uvm_ublk::ctrl: begin to stop_dev dev_id=224
2026-08-14T10:02:06.713241Z DEBUG uvm_ublk::dev: submit and fetch for slot failed tag=0 dev_id=224 err=Os { code: 19, kind: Uncategorized, message: "No such device" }
2026-08-14T10:02:06.713283Z DEBUG uvm_ublk::dev: submit and fetch for slot failed tag=1 dev_id=224 err=Os { code: 19, kind: Uncategorized, message: "No such device" }
2026-08-14T10:02:06.713446Z  INFO uvm_ublk::dev: queue slot task finished err=submit and fetch for slot from ring failed: Os { code: 19, kind: Uncategorized, message: "No such device" }
2026-08-14T10:02:06.713450Z  INFO uvm_ublk::dev: queue slot task finished err=submit and fetch for slot from ring failed: Os { code: 19, kind: Uncategorized, message: "No such device" }
2026-08-14T10:02:06.786851Z  INFO uvm_ublk::ctrl: stop_dev ctrl cmd completed dev_id=224 res=0
2026-08-14T10:02:06.786886Z DEBUG uvm_ublk::dev: UVMUblkDevInner has dropped


memleak-bpfcc


# Allocation Hotspot 1: Unreleased RawVec dynamic growth
27525120 bytes in 59 allocations from stack
        0x0000aaaad074e6d4      <alloc::raw_vec::RawVecInner>::finish_grow+0x5c [uvm-ublk-daemon]
        0x0000aaaad074e5e4      <alloc::raw_vec::RawVec<(std::path::PathBuf, u64, std::time::SystemTime)>>::grow_one+0x3c [uvm-ublk-daemon]
        0x0000aaaad0b56c58      overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}+0x249c [uvm-ublk-daemon]
        0x0000aaaad0b5ca74      <tokio::runtime::task::core::Core<overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll+0x3c [uvm-ublk-daemon]
        0x0000aaaad0b4294c      <tokio::runtime::task::harness::Harness<overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll+0x54 [uvm-ublk-daemon]
        ...

# Allocation Hotspot 2: Directory iteration via tokio::fs::read_dir
7179023 bytes in 101113 allocations from stack
        0x0000aaaad118a16c      <core::wtf8::Wtf8>::into_box+0x2c [uvm-ublk-daemon]
        0x0000aaaad117baf0      <std::sys::fs::unix::ReadDir as core::iter::traits::iterator::Iterator>::next+0xb4 [uvm-ublk-daemon]
        0x0000aaaad117e658      <std::fs::ReadDir as core::iter::traits::iterator::Iterator>::next+0x1c [uvm-ublk-daemon]
        0x0000aaaad10f8764      <tokio::fs::read_dir::ReadDir>::next_chunk+0x78 [uvm-ublk-daemon]
        0x0000aaaad10fb424      <tokio::runtime::blocking::task::BlockingTask<<tokio::fs::read_dir::ReadDir>::poll_next_entry::{closure#0}> as core::future::future::Future>::poll+0xac [uvm-ublk-daemon]
        ...

Reproduction frequency

Always

Regression information

Last known good:
First known bad:

Additional context

No response

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @YXalix on GitHub (Aug 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/171 ### Affected component ublk / ublk-daemon ### AgentENV version - Version or commit:0475f403b119c29d3b74aa32b5c10dff07c68493 - Local modifications: none ### Host environment - OS:debian(velinux) - Kernel (`uname -a`):6.6.95 - Architecture:ARM64 - Bare metal, VM, or cloud:Bare metal - ublk support:ublk_drv loaded - CPU:kunpeng 950 - Memory: 1.5T - Storage/filesystem:ext4 ### Operation and configuration ```shell During stress testing involving 600 virtual machines continuously executing pause and resume cycles, the uvm-ublk-daemon process exhibits rapid and uncontrolled memory growth in anon_mmap at a rate of 20–40 MB/s, eventually reaching tens of gigabytes. Memory allocations captured using memleak-bpfcc indicate that the majority of unreleased memory stems from overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write and tokio::fs::read_dir. ``` ### Steps to reproduce 1. Sequentially launch 600 virtual machines: - Keep 100 VMs in the running state. - Keep the remaining 500 VMs in the pause state. 2. Run a continuous loop of pause + resume operations across all 600 VMs. 3. Observe the memory consumption metrics of uvm-ublk-daemon. ### Expected behavior uvm-ublk-daemon memory consumption should remain stable within a bounded range during repeated VM pause/resume cycles. ### Actual behavior anon_mmap grows linearly at 20–40 MB/s. ### Logs and diagnostics ```text aenv/logs/ublk-daemon.log 2026-08-14T10:02:06.598578Z INFO uvm_ublk_daemon::server: ReleaseOverlaybd { dev_id: 224 } 2026-08-14T10:02:06.598602Z INFO uvm_ublk_daemon::server: releasing shared device (refcount reached 0) dev_id=224 2026-08-14T10:02:06.598625Z DEBUG uvm_ublk_daemon::server: cleared page cache path=/dev/ublkb224 2026-08-14T10:02:06.599138Z INFO uvm_ublk_daemon::server: idle overlaybd pool is full; stopping returned device dev_id=224 high_watermark=128 2026-08-14T10:02:06.599144Z INFO uvm_ublk_daemon::server: stopping pooled overlaybd device during shutdown dev_id=224 2026-08-14T10:02:06.599149Z DEBUG uvm_ublk::ctrl: begin to stop_dev dev_id=224 2026-08-14T10:02:06.713241Z DEBUG uvm_ublk::dev: submit and fetch for slot failed tag=0 dev_id=224 err=Os { code: 19, kind: Uncategorized, message: "No such device" } 2026-08-14T10:02:06.713283Z DEBUG uvm_ublk::dev: submit and fetch for slot failed tag=1 dev_id=224 err=Os { code: 19, kind: Uncategorized, message: "No such device" } 2026-08-14T10:02:06.713446Z INFO uvm_ublk::dev: queue slot task finished err=submit and fetch for slot from ring failed: Os { code: 19, kind: Uncategorized, message: "No such device" } 2026-08-14T10:02:06.713450Z INFO uvm_ublk::dev: queue slot task finished err=submit and fetch for slot from ring failed: Os { code: 19, kind: Uncategorized, message: "No such device" } 2026-08-14T10:02:06.786851Z INFO uvm_ublk::ctrl: stop_dev ctrl cmd completed dev_id=224 res=0 2026-08-14T10:02:06.786886Z DEBUG uvm_ublk::dev: UVMUblkDevInner has dropped memleak-bpfcc # Allocation Hotspot 1: Unreleased RawVec dynamic growth 27525120 bytes in 59 allocations from stack 0x0000aaaad074e6d4 <alloc::raw_vec::RawVecInner>::finish_grow+0x5c [uvm-ublk-daemon] 0x0000aaaad074e5e4 <alloc::raw_vec::RawVec<(std::path::PathBuf, u64, std::time::SystemTime)>>::grow_one+0x3c [uvm-ublk-daemon] 0x0000aaaad0b56c58 overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}+0x249c [uvm-ublk-daemon] 0x0000aaaad0b5ca74 <tokio::runtime::task::core::Core<overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll+0x3c [uvm-ublk-daemon] 0x0000aaaad0b4294c <tokio::runtime::task::harness::Harness<overlaybd::lsmt::file::helper::spawn_premerged_index_artifact_write::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll+0x54 [uvm-ublk-daemon] ... # Allocation Hotspot 2: Directory iteration via tokio::fs::read_dir 7179023 bytes in 101113 allocations from stack 0x0000aaaad118a16c <core::wtf8::Wtf8>::into_box+0x2c [uvm-ublk-daemon] 0x0000aaaad117baf0 <std::sys::fs::unix::ReadDir as core::iter::traits::iterator::Iterator>::next+0xb4 [uvm-ublk-daemon] 0x0000aaaad117e658 <std::fs::ReadDir as core::iter::traits::iterator::Iterator>::next+0x1c [uvm-ublk-daemon] 0x0000aaaad10f8764 <tokio::fs::read_dir::ReadDir>::next_chunk+0x78 [uvm-ublk-daemon] 0x0000aaaad10fb424 <tokio::runtime::blocking::task::BlockingTask<<tokio::fs::read_dir::ReadDir>::poll_next_entry::{closure#0}> as core::future::future::Future>::poll+0xac [uvm-ublk-daemon] ... ``` ### Reproduction frequency Always ### Regression information - Last known good: - First known bad: ### Additional context _No response_ ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:18:56 +00:00