deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #79] fio userbio plugin run failed, Segmentation fault #44

New issue
Closed
opened 2026-10-26 09:44:00 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-26 09:44:00 +00:00
Owner
Copy link

Originally created by @zhangke96 on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/79

Originally assigned to: @KuribohG on GitHub.

I run fio with ioengine hf3fs_usrbio failed.
build commit c3a16b5cd8caf7a604270813cc4a5a0772b3d279
fs mount info

hf3fs.set6666   336T  3.2T  333T   1% /mnt
hf3fs.set6666 on /mnt type fuse.hf3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=1048576)

fio version

fio-3.28

fio command

fio -ioengine=external:hf3fs_usrbio.so -directory=/mnt/fio_test -name=test -size=1G -rw=read -bs=1M -numjobs=1  --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0

fio output

test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=hf3fs_usrbio, iodepth=1
fio-3.28
Starting 1 process
fio: io engine hf3fs_usrbio init failed.
fio: pid=597668, err=-22/
fio: pid=597668, got signal=11

coredump info

#0  __gnu_cxx::__exchange_and_add (__mem=0x7fe318cd, __val=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:66
66	  { return __atomic_fetch_add(__mem, __val, __ATOMIC_ACQ_REL); }
(gdb) bt
#0  __gnu_cxx::__exchange_and_add (__mem=0x7fe318cd, __val=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:66
#1  __gnu_cxx::__exchange_and_add_dispatch (__mem=0x7fe318cd, __val=-7056) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:101
#2  folly::detail::shared_ptr_internals::release_shared<hf3fs::storage::client::IOBuffer> (base=0x7fe318c5, count=7056) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/detail/AtomicSharedPtr-detail.h:157
#3  folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::release_external (c=0, res=...) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:250
#4  folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::store (n=std::shared_ptr<hf3fs::storage::client::IOBuffer> (empty) = {...}, order=std::memory_order::seq_cst, 
    this=<optimized out>) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:119
#5  folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::~atomic_shared_ptr (this=this@entry=0x7fe318c54480)
    at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:90
#6  0x00007fe30e77a0a8 in std::destroy_at<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__location=0x7fe318c54480)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:88
#7  std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__pointer=0x7fe318c54480)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:149
#8  std::_Destroy_aux<false>::__destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*> (__first=0x7fe318c54480, __last=0x7fe318c54490)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:163
#9  std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*> (__first=<optimized out>, __last=0x7fe318c54490)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:195
#10 std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*, folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__first=<optimized out>, __last=0x7fe318c54490) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/alloc_traits.h:850
#11 std::vector<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>, std::allocator<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > >::~vector (this=0x7fe318c54498) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:730
#12 hf3fs::lib::ShmBuf::~ShmBuf (this=0x7fe318c543e0) at /home/ubuntu/3FS/src/lib/common/Shm.cc:65
#13 0x00007fe30e837e0d in hf3fs_iovdestroy_general (iov=0x55c16c4c0490, is_io_ring=false, for_read=<optimized out>, io_depth=0, priority=0, timeout=0, flags=<optimized out>) at /home/ubuntu/3FS/src/lib/api/UsrbIo.cc:218
#14 0x00007fe30e838cfd in hf3fs_iovdestroy (iov=0x7fffad299c30) at /home/ubuntu/3FS/src/lib/api/UsrbIo.cc:288
#15 0x000055c16c1a3fbb in free_io_mem ()
#16 0x000055c16c1d7c1d in ?? ()
#17 0x000055c16c1db578 in ?? ()
#18 0x000055c16c1db690 in fio_backend ()
#19 0x000055c16c17e301 in main ()
Originally created by @zhangke96 on GitHub (Mar 4, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/79 Originally assigned to: @KuribohG on GitHub. I run fio with ioengine hf3fs_usrbio failed. build commit [c3a16b5cd8caf7a604270813cc4a5a0772b3d279](https://github.com/deepseek-ai/3FS/commit/c3a16b5cd8caf7a604270813cc4a5a0772b3d279) fs mount info ``` hf3fs.set6666 336T 3.2T 333T 1% /mnt hf3fs.set6666 on /mnt type fuse.hf3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=1048576) ``` fio version ``` fio-3.28 ``` fio command ``` fio -ioengine=external:hf3fs_usrbio.so -directory=/mnt/fio_test -name=test -size=1G -rw=read -bs=1M -numjobs=1 --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0 ``` fio output ``` test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=hf3fs_usrbio, iodepth=1 fio-3.28 Starting 1 process fio: io engine hf3fs_usrbio init failed. fio: pid=597668, err=-22/ fio: pid=597668, got signal=11 ``` coredump info ``` #0 __gnu_cxx::__exchange_and_add (__mem=0x7fe318cd, __val=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:66 66 { return __atomic_fetch_add(__mem, __val, __ATOMIC_ACQ_REL); } (gdb) bt #0 __gnu_cxx::__exchange_and_add (__mem=0x7fe318cd, __val=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:66 #1 __gnu_cxx::__exchange_and_add_dispatch (__mem=0x7fe318cd, __val=-7056) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/ext/atomicity.h:101 #2 folly::detail::shared_ptr_internals::release_shared<hf3fs::storage::client::IOBuffer> (base=0x7fe318c5, count=7056) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/detail/AtomicSharedPtr-detail.h:157 #3 folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::release_external (c=0, res=...) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:250 #4 folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::store (n=std::shared_ptr<hf3fs::storage::client::IOBuffer> (empty) = {...}, order=std::memory_order::seq_cst, this=<optimized out>) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:119 #5 folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>::~atomic_shared_ptr (this=this@entry=0x7fe318c54480) at /home/ubuntu/3FS/third_party/folly/folly/concurrency/AtomicSharedPtr.h:90 #6 0x00007fe30e77a0a8 in std::destroy_at<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__location=0x7fe318c54480) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:88 #7 std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__pointer=0x7fe318c54480) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:149 #8 std::_Destroy_aux<false>::__destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*> (__first=0x7fe318c54480, __last=0x7fe318c54490) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:163 #9 std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*> (__first=<optimized out>, __last=0x7fe318c54490) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_construct.h:195 #10 std::_Destroy<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>*, folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > (__first=<optimized out>, __last=0x7fe318c54490) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/alloc_traits.h:850 #11 std::vector<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals>, std::allocator<folly::atomic_shared_ptr<hf3fs::storage::client::IOBuffer, std::atomic, folly::detail::shared_ptr_internals> > >::~vector (this=0x7fe318c54498) at /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:730 #12 hf3fs::lib::ShmBuf::~ShmBuf (this=0x7fe318c543e0) at /home/ubuntu/3FS/src/lib/common/Shm.cc:65 #13 0x00007fe30e837e0d in hf3fs_iovdestroy_general (iov=0x55c16c4c0490, is_io_ring=false, for_read=<optimized out>, io_depth=0, priority=0, timeout=0, flags=<optimized out>) at /home/ubuntu/3FS/src/lib/api/UsrbIo.cc:218 #14 0x00007fe30e838cfd in hf3fs_iovdestroy (iov=0x7fffad299c30) at /home/ubuntu/3FS/src/lib/api/UsrbIo.cc:288 #15 0x000055c16c1a3fbb in free_io_mem () #16 0x000055c16c1d7c1d in ?? () #17 0x000055c16c1db578 in ?? () #18 0x000055c16c1db690 in fio_backend () #19 0x000055c16c17e301 in main () ```
bsakaguchi commented 2026-10-26 09:44:02 +00:00
Author
Owner
Copy link

@beanwang24 commented on GitHub (Mar 4, 2025):

I'm encountering the same issue, the error message shows:
fio: io engine hf3fs_usrbio init failed.
fio:3471854 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap()

<!-- gh-comment-id:2696342993 --> @beanwang24 commented on GitHub (Mar 4, 2025): I'm encountering the same issue, the error message shows: fio: io engine hf3fs_usrbio init failed. fio:3471854 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap()
bsakaguchi commented 2026-10-26 09:44:02 +00:00
Author
Owner
Copy link

@limengzhechen commented on GitHub (Mar 4, 2025):

hi, how to build external:hf3fs_usrbio.so?

root@li:/home/li/fio-master# cd /home/li/fio-master
root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/
CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile
root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/
CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md
root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./

then, i can't find hf3fs_usrbio.so ?

<!-- gh-comment-id:2696463497 --> @limengzhechen commented on GitHub (Mar 4, 2025): hi, how to build external:hf3fs_usrbio.so? root@li:/home/li/fio-master# cd /home/li/fio-master root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/ CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/ CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./ then, i can't find hf3fs_usrbio.so ?
bsakaguchi commented 2026-10-26 09:44:03 +00:00
Author
Owner
Copy link

@zhangke96 commented on GitHub (Mar 4, 2025):

hi, how to build external:hf3fs_usrbio.so?

root@li:/home/li/fio-master# cd /home/li/fio-master root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/ CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/ CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./

then, i can't find hf3fs_usrbio.so ?
first, build fio, then run make in 3FS/benchmarks/fio_usrbio directory

<!-- gh-comment-id:2696477578 --> @zhangke96 commented on GitHub (Mar 4, 2025): > hi, how to build external:hf3fs_usrbio.so? > > root@li:/home/li/fio-master# cd /home/li/fio-master root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/ CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/ CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./ > > then, i can't find hf3fs_usrbio.so ? first, build fio, then run make in `3FS/benchmarks/fio_usrbio` directory
bsakaguchi commented 2026-10-26 09:44:03 +00:00
Author
Owner
Copy link

@zhangke96 commented on GitHub (Mar 4, 2025):

hi, how to build external:hf3fs_usrbio.so?
root@li:/home/li/fio-master# cd /home/li/fio-master root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/ CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/ CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./
then, i can't find hf3fs_usrbio.so ?
first, build fio, then run make in 3FS/benchmarks/fio_usrbio directory

I'm have build issue

/home/ubuntu/ubuntu_fio/fio-3.28/os/os-linux.h:351:23: error: conflicting declaration of C function ‘ssize_t pwritev2(int, const iovec*, int, off_t, unsigned int)’
  351 | static inline ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt,
      |                       ^~~~~~~~
In file included from /home/ubuntu/ubuntu_fio/fio-3.28/os/os-linux.h:7,
                 from /home/ubuntu/ubuntu_fio/fio-3.28/os/os.h:39,
                 from /home/ubuntu/ubuntu_fio/fio-3.28/thread_options.h:5,
                 from /home/ubuntu/ubuntu_fio/fio-3.28/fio.h:18,
                 from hf3fs_usrbio.cpp:16:
/usr/include/x86_64-linux-gnu/sys/uio.h:138:16: note: previous declaration ‘ssize_t pwritev2(int, const iovec*, int, __off_t, int)’
  138 | extern ssize_t pwritev2 (int __fd, const struct iovec *__iodev, int __count,
      |

I try to add DEBUG macro with g++ cmd -DCONFIG_SYNC_FILE_RANGE -DCONFIG_PWRITEV2 -DCONFIG_PREADV2 to solve.

<!-- gh-comment-id:2696482958 --> @zhangke96 commented on GitHub (Mar 4, 2025): > > hi, how to build external:hf3fs_usrbio.so? > > root@li:/home/li/fio-master# cd /home/li/fio-master root@li:/home/li/fio-master# ls /home/li/3FS/build/src/lib/api/ CMakeFiles cmake_install.cmake CTestTestfile.cmake libhf3fs_api.a libhf3fs_api_shared.so Makefile root@li:/home/li/fio-master# ls /home/li/3FS/src/lib/api/ CMakeLists.txt fuse.h hf3fs_expected.h hf3fs.h hf3fs_usrbio.h UsrbIo.cc UsrbIo.md root@li:/home/li/fio-master# make HF3FS_LIB_DIR=/home/li/3FS/build/src/lib/api/ HF3FS_INCLUDE_DIR=/home/li/3FS/src/lib/api/ FIO_SRC_DIR=./ > > then, i can't find hf3fs_usrbio.so ? > > first, build fio, then run make in `3FS/benchmarks/fio_usrbio` directory I'm have build issue ``` /home/ubuntu/ubuntu_fio/fio-3.28/os/os-linux.h:351:23: error: conflicting declaration of C function ‘ssize_t pwritev2(int, const iovec*, int, off_t, unsigned int)’ 351 | static inline ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, | ^~~~~~~~ In file included from /home/ubuntu/ubuntu_fio/fio-3.28/os/os-linux.h:7, from /home/ubuntu/ubuntu_fio/fio-3.28/os/os.h:39, from /home/ubuntu/ubuntu_fio/fio-3.28/thread_options.h:5, from /home/ubuntu/ubuntu_fio/fio-3.28/fio.h:18, from hf3fs_usrbio.cpp:16: /usr/include/x86_64-linux-gnu/sys/uio.h:138:16: note: previous declaration ‘ssize_t pwritev2(int, const iovec*, int, __off_t, int)’ 138 | extern ssize_t pwritev2 (int __fd, const struct iovec *__iodev, int __count, | ``` I try to add DEBUG macro with g++ cmd `-DCONFIG_SYNC_FILE_RANGE -DCONFIG_PWRITEV2 -DCONFIG_PREADV2` to solve.
bsakaguchi commented 2026-10-26 09:44:03 +00:00
Author
Owner
Copy link

@limengzhechen commented on GitHub (Mar 4, 2025):

i use fio-3.39, also fail like:
./fio -ioengine=external:hf3fs_usrbio.so -directory=//3fs/stage/dir/file-1G -name=test -size=1G -rw=read -bs=1M -numjobs=1 --group_reporting -time_based=1 -runtime=120 -iodepth=1 -direct=0
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=hf3fs_usrbio, iodepth=1
fio-3.39
Starting 1 process
fio: io engine hf3fs_usrbio init failed.
fio: pid=692813, err=-22/
[2025-03-04T07:59:24.581448888+00:00 fio:692813 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap()

Run status group 0 (all jobs):

<!-- gh-comment-id:2696564322 --> @limengzhechen commented on GitHub (Mar 4, 2025): i use fio-3.39, also fail like: ./fio -ioengine=external:hf3fs_usrbio.so -directory=//3fs/stage/dir/file-1G -name=test -size=1G -rw=read -bs=1M -numjobs=1 --group_reporting -time_based=1 -runtime=120 -iodepth=1 -direct=0 test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=hf3fs_usrbio, iodepth=1 fio-3.39 Starting 1 process fio: io engine hf3fs_usrbio init failed. fio: pid=692813, err=-22/ [2025-03-04T07:59:24.581448888+00:00 fio:692813 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap() Run status group 0 (all jobs):
bsakaguchi commented 2026-10-26 09:44:04 +00:00
Author
Owner
Copy link

@KuribohG commented on GitHub (Mar 4, 2025):

Sorry, we forgot to add one extra parameter needed in our USRBIO fio plugin.
mountpoint is needed in fio configure file, for your instance you can add mountpoint=/mnt to solve this problem.
I updated the README.md for this.

<!-- gh-comment-id:2696636827 --> @KuribohG commented on GitHub (Mar 4, 2025): Sorry, we forgot to add one extra parameter needed in our USRBIO fio plugin. `mountpoint` is needed in fio configure file, for your instance you can add `mountpoint=/mnt` to solve this problem. I updated the [README.md](https://github.com/deepseek-ai/3FS/tree/main/benchmarks/fio_usrbio) for this.
bsakaguchi commented 2026-10-26 09:44:04 +00:00
Author
Owner
Copy link

@limengzhechen commented on GitHub (Mar 4, 2025):

./fio -ioengine=external:hf3fs_usrbio.so -directory=/3fs/stage/dir/file-1G/ -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0 -mountpoint=/3fs/stage/dir/file-1G
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=hf3fs_usrbio, iodepth=1
fio-3.39
Starting 1 process
[2025-03-04T08:40:39.431837026+00:00 fio:712318 UsrbIo.cc:155 ERROR] failed to register iov '/dev/shm/hf3fs-iov-e4c77afb-b03b-47b6-ac0d-b1563305a3df' to hf3fs '/3fs/stage/dir/file-1G/3fs-virt/iovs/7e771695-daef-4119-8baf-14210dd0ab2b.r0.t1'
[2025-03-04T08:40:39.432040691+00:00 fio:712318 UsrbIo.cc:463 ERROR] ioring create failed: hf3fs_iovcreate_general failed -2
fio: io engine hf3fs_usrbio init failed.
fio: pid=712318, err=-2/
[2025-03-04T08:40:39.432088940+00:00 fio:712318 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap()

Run status group 0 (all jobs):

<!-- gh-comment-id:2696714304 --> @limengzhechen commented on GitHub (Mar 4, 2025): ./fio -ioengine=external:hf3fs_usrbio.so -directory=/3fs/stage/dir/file-1G/ -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0 -mountpoint=/3fs/stage/dir/file-1G test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=hf3fs_usrbio, iodepth=1 fio-3.39 Starting 1 process [2025-03-04T08:40:39.431837026+00:00 fio:712318 UsrbIo.cc:155 ERROR] failed to register iov '/dev/shm/hf3fs-iov-e4c77afb-b03b-47b6-ac0d-b1563305a3df' to hf3fs '/3fs/stage/dir/file-1G/3fs-virt/iovs/7e771695-daef-4119-8baf-14210dd0ab2b.r0.t1' [2025-03-04T08:40:39.432040691+00:00 fio:712318 UsrbIo.cc:463 ERROR] ioring create failed: hf3fs_iovcreate_general failed -2 fio: io engine hf3fs_usrbio init failed. fio: pid=712318, err=-2/ [2025-03-04T08:40:39.432088940+00:00 fio:712318 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap() Run status group 0 (all jobs):
bsakaguchi commented 2026-10-26 09:44:04 +00:00
Author
Owner
Copy link

@zhangke96 commented on GitHub (Mar 4, 2025):

Sorry, we forgot to add one extra parameter needed in our USRBIO fio plugin. mountpoint is needed in fio configure file, for your instance you can add mountpoint=/mnt to solve this problem. I updated the README.md for this.

It works. The client cpu usage has been greatly reduced.

<!-- gh-comment-id:2696788199 --> @zhangke96 commented on GitHub (Mar 4, 2025): > Sorry, we forgot to add one extra parameter needed in our USRBIO fio plugin. `mountpoint` is needed in fio configure file, for your instance you can add `mountpoint=/mnt` to solve this problem. I updated the [README.md](https://github.com/deepseek-ai/3FS/tree/main/benchmarks/fio_usrbio) for this. It works. The client cpu usage has been greatly reduced.
bsakaguchi commented 2026-10-26 09:44:04 +00:00
Author
Owner
Copy link

@zhangke96 commented on GitHub (Mar 4, 2025):

./fio -ioengine=external:hf3fs_usrbio.so -directory=/3fs/stage/dir/file-1G/ -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0 -mountpoint=/3fs/stage/dir/file-1G test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=hf3fs_usrbio, iodepth=1 fio-3.39 Starting 1 process [2025-03-04T08:40:39.431837026+00:00 fio:712318 UsrbIo.cc:155 ERROR] failed to register iov '/dev/shm/hf3fs-iov-e4c77afb-b03b-47b6-ac0d-b1563305a3df' to hf3fs '/3fs/stage/dir/file-1G/3fs-virt/iovs/7e771695-daef-4119-8baf-14210dd0ab2b.r0.t1' [2025-03-04T08:40:39.432040691+00:00 fio:712318 UsrbIo.cc:463 ERROR] ioring create failed: hf3fs_iovcreate_general failed -2 fio: io engine hf3fs_usrbio init failed. fio: pid=712318, err=-2/ [2025-03-04T08:40:39.432088940+00:00 fio:712318 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap()

Run status group 0 (all jobs):

-mountpoint=/3fs/stage/dir/file-1G change to -mountpoint=$mount_root_dir

<!-- gh-comment-id:2696795254 --> @zhangke96 commented on GitHub (Mar 4, 2025): > ./fio -ioengine=external:hf3fs_usrbio.so -directory=/3fs/stage/dir/file-1G/ -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=1000 -iodepth=1 -direct=0 -mountpoint=/3fs/stage/dir/file-1G test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=hf3fs_usrbio, iodepth=1 fio-3.39 Starting 1 process [2025-03-04T08:40:39.431837026+00:00 fio:712318 UsrbIo.cc:155 ERROR] failed to register iov '/dev/shm/hf3fs-iov-e4c77afb-b03b-47b6-ac0d-b1563305a3df' to hf3fs '/3fs/stage/dir/file-1G/3fs-virt/iovs/7e771695-daef-4119-8baf-14210dd0ab2b.r0.t1' [2025-03-04T08:40:39.432040691+00:00 fio:712318 UsrbIo.cc:463 ERROR] ioring create failed: hf3fs_iovcreate_general failed -2 fio: io engine hf3fs_usrbio init failed. fio: pid=712318, err=-2/ [2025-03-04T08:40:39.432088940+00:00 fio:712318 UsrbIo.cc:196 ERROR] cannot iovfree() an iov created by iovwrap() > > Run status group 0 (all jobs): `-mountpoint=/3fs/stage/dir/file-1G` change to `-mountpoint=$mount_root_dir`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#44
No description provided.