deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #323] 无法注册文件 #199

New issue
Closed
opened 2026-10-26 09:47:02 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:47:02 +00:00
Owner
Copy link

Originally created by @skyCreateXian on GitHub (Aug 20, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/323

[2025-08-20 12:50:18 TP1] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP1] [Rank 1] HiCacheHF3FS Client Initializing: file_path=/data/hicache.1.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP6] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP6] [Rank 6] HiCacheHF3FS Client Initializing: file_path=/data/hicache.6.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP6] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP0] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP0] [Rank 0] HiCacheHF3FS Client Initializing: file_path=/data/hicache.0.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP2] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP2] [Rank 2] HiCacheHF3FS Client Initializing: file_path=/data/hicache.2.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP4] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP4] [Rank 4] HiCacheHF3FS Client Initializing: file_path=/data/hicache.4.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP3] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP3] [Rank 3] HiCacheHF3FS Client Initializing: file_path=/data/hicache.3.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP5] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP5] [Rank 5] HiCacheHF3FS Client Initializing: file_path=/data/hicache.5.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP5] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 12:50:18 TP7] Using global metadata client with server url: http://127.0.0.1:18000
[2025-08-20 12:50:18 TP7] [Rank 7] HiCacheHF3FS Client Initializing: file_path=/data/hicache.7.bin, file_size=240.00 GB, num_pages=57299
[2025-08-20 12:50:18 TP7] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__
    self.init_memory_pool_and_cache()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache
    self.tree_cache = HiRadixCache(
                      ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__
    self.cache_controller = HiCacheController(
                            ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__
    self.storage_backend = HiCacheHF3FS.from_env_config(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config
    return HiCacheHF3FS(
           ^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__
    Hf3fsClient(
  File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__
    register_fd(self.file)
OSError: [Errno 9] Bad file descriptor

[2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed.

我在sglang上让3fs作为L3 cache时发生crash，似乎文件无法写入

Originally created by @skyCreateXian on GitHub (Aug 20, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/323 ``` [2025-08-20 12:50:18 TP1] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP1] [Rank 1] HiCacheHF3FS Client Initializing: file_path=/data/hicache.1.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP6] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP6] [Rank 6] HiCacheHF3FS Client Initializing: file_path=/data/hicache.6.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP6] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP0] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP0] [Rank 0] HiCacheHF3FS Client Initializing: file_path=/data/hicache.0.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP2] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP2] [Rank 2] HiCacheHF3FS Client Initializing: file_path=/data/hicache.2.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP2] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP4] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP4] [Rank 4] HiCacheHF3FS Client Initializing: file_path=/data/hicache.4.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP4] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP3] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP3] [Rank 3] HiCacheHF3FS Client Initializing: file_path=/data/hicache.3.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP5] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP5] [Rank 5] HiCacheHF3FS Client Initializing: file_path=/data/hicache.5.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP5] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. [2025-08-20 12:50:18 TP7] Using global metadata client with server url: http://127.0.0.1:18000 [2025-08-20 12:50:18 TP7] [Rank 7] HiCacheHF3FS Client Initializing: file_path=/data/hicache.7.bin, file_size=240.00 GB, num_pages=57299 [2025-08-20 12:50:18 TP7] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 401, in __init__ self.init_memory_pool_and_cache() File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 607, in init_memory_pool_and_cache self.tree_cache = HiRadixCache( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 80, in __init__ self.cache_controller = HiCacheController( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 275, in __init__ self.storage_backend = HiCacheHF3FS.from_env_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 219, in from_env_config return HiCacheHF3FS( ^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/storage_hf3fs.py", line 150, in __init__ Hf3fsClient( File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/hf3fs/client_hf3fs.py", line 68, in __init__ register_fd(self.file) OSError: [Errno 9] Bad file descriptor [2025-08-20 12:50:18] Received sigquit from a child process. It usually means the child failed. ``` 我在sglang上让3fs作为L3 cache时发生crash，似乎文件无法写入
bsakaguchi commented 2026-10-26 09:47:04 +00:00
Author
Owner
Copy link

@skyCreateXian commented on GitHub (Aug 20, 2025):

文件系统容量是足够的

# df -h
Filesystem                Size  Used Avail Use% Mounted on
overlay                   3.0T  909G  2.2T  30% /
<!-- gh-comment-id:3206219087 --> @skyCreateXian commented on GitHub (Aug 20, 2025): 文件系统容量是足够的 ``` # df -h Filesystem Size Used Avail Use% Mounted on overlay 3.0T 909G 2.2T 30% / ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#199
No description provided.