deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #160] 同样的测试方式下，fio直读时延在微秒级，但是直写平均时延有毫秒级，如何降低写时延呢？ #96

New issue
Closed
opened 2026-10-26 09:45:01 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:45:01 +00:00
Owner
Copy link

Originally created by @222luhang on GitHub (Mar 12, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/160

在默认配置下进行的fio测试(depth修改32-->1)
读

fio -ioengine=libaio -directory=/mnt/fio_test -name=test -size=1G -rw=read -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=300 -iodepth=1 -direct=1

test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=6556KiB/s][r=1639 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=736789: Wed Mar 12 10:25:26 2025
  read: IOPS=1475, BW=5902KiB/s (6044kB/s)(1729MiB/300001msec)
    slat (nsec): min=4722, max=163445, avg=6706.66, stdev=882.60
    clat (usec): min=399, max=15810, avg=669.57, stdev=109.33
     lat (usec): min=407, max=15816, avg=676.53, stdev=109.36
    clat percentiles (usec):
     |  1.00th=[  529],  5.00th=[  545], 10.00th=[  553], 20.00th=[  562],
     | 30.00th=[  578], 40.00th=[  586], 50.00th=[  660], 60.00th=[  725],
     | 70.00th=[  750], 80.00th=[  783], 90.00th=[  816], 95.00th=[  832],
     | 99.00th=[  865], 99.50th=[  873], 99.90th=[  922], 99.95th=[  955],
     | 99.99th=[ 1074]
   bw (  KiB/s): min= 4968, max= 7088, per=100.00%, avg=5902.88, stdev=445.03, samples=599
   iops        : min= 1242, max= 1772, avg=1475.72, stdev=111.26, samples=599
  lat (usec)   : 500=0.05%, 750=71.24%, 1000=28.68%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.54%, sys=1.67%, ctx=442648, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=442646,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5902KiB/s (6044kB/s), 5902KiB/s-5902KiB/s (6044kB/s-6044kB/s), io=1729MiB (1813MB), run=300001-300001msec

写

fio -ioengine=libaio -directory=/mnt/fio_test -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=300 -iodepth=1 -direct=1

test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1388KiB/s][w=347 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=736706: Wed Mar 12 10:18:15 2025
  write: IOPS=264, BW=1057KiB/s (1082kB/s)(310MiB/300006msec); 0 zone resets
    slat (usec): min=2, max=259, avg= 6.17, stdev= 1.08
    clat (usec): min=1059, max=16990, avg=3777.01, stdev=1759.73
     lat (usec): min=1066, max=16996, avg=3783.48, stdev=1759.79
    clat percentiles (usec):
     |  1.00th=[ 1287],  5.00th=[ 1303], 10.00th=[ 1303], 20.00th=[ 1319],
     | 30.00th=[ 1549], 40.00th=[ 3982], 50.00th=[ 4293], 60.00th=[ 4555],
     | 70.00th=[ 4883], 80.00th=[ 5276], 90.00th=[ 5800], 95.00th=[ 6128],
     | 99.00th=[ 6652], 99.50th=[ 6849], 99.90th=[ 7373], 99.95th=[ 8291],
     | 99.99th=[15926]
   bw (  KiB/s): min=  696, max= 3024, per=100.00%, avg=1057.50, stdev=507.82, samples=599
   iops        : min=  174, max=  756, avg=264.37, stdev=126.95, samples=599
  lat (msec)   : 2=30.25%, 4=10.00%, 10=59.72%, 20=0.03%
  cpu          : usr=0.12%, sys=0.27%, ctx=79267, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,79265,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1057KiB/s (1082kB/s), 1057KiB/s-1057KiB/s (1082kB/s-1082kB/s), io=310MiB (325MB), run=300006-300006msec

客户端日志中的时延统计，耗时最多在submit

hf3fs_fuse_main.1.log:[2025-03-12T10:15:57.977300439+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8072 mean:4us min:1us max:18us p50:4us p90:5us p95:5us p99:6us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.977022018+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8127 mean:4us min:1us max:18us p50:4us p90:5us p95:5us p99:7us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.977352049+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8060 mean:4us min:3us max:32us p50:4us p90:5us p95:5us p99:8us
hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.977142518+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7824 mean:4us min:1us max:37us p50:4us p90:5us p95:5us p99:6us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.973805710+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:2749 mean:5us min:2us max:32us p50:4us p90:6us p95:8us p99:9us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.974500515+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7807 mean:5us min:2us max:29us p50:4us p90:7us p95:8us p99:9us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:25.974563626+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8164 mean:5us min:1us max:40us p50:4us p90:7us p95:8us p99:9us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:55.977177490+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7479 mean:4us min:2us max:69us p50:4us p90:5us p95:6us p99:8us
hf3fs_fuse_main.2.log:[2025-03-12T10:15:26.977794989+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8700 mean:4us min:3us max:19us p50:4us p90:5us p95:5us p99:6us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.978103726+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:9864 mean:5us min:1us max:36us p50:5us p90:6us p95:6us p99:7us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.978654351+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10063 mean:5us min:3us max:51us p50:5us p90:6us p95:6us p99:8us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.977743566+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10110 mean:5us min:3us max:41us p50:5us p90:6us p95:6us p99:7us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.975045407+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:6622 mean:5us min:1us max:40us p50:5us p90:6us p95:6us p99:8us
hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.974779973+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10026 mean:5us min:3us max:41us p50:5us p90:6us p95:6us p99:8us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.979509807+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:9859 mean:5us min:1us max:97us p50:5us p90:6us p95:6us p99:8us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.980144439+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10122 mean:5us min:1us max:44us p50:5us p90:6us p95:6us p99:7us
hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.974228808+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:262 mean:7us min:3us max:61us p50:5us p90:12us p95:14us p99:39us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.979397655+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10110 mean:5us min:2us max:31us p50:5us p90:6us p95:7us p99:9us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.979719352+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10056 mean:5us min:3us max:48us p50:5us p90:6us p95:7us p99:9us
hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.979134929+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10086 mean:5us min:1us max:75us p50:5us p90:6us p95:6us p99:8us
hf3fs_fuse_main.log:[2025-03-12T10:17:59.977448568+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8299 mean:4us min:1us max:23us p50:4us p90:5us p95:5us p99:7us
hf3fs_fuse_main.log:[2025-03-12T10:18:30.974932857+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:3984 mean:4us min:2us max:40us p50:4us p90:5us p95:5us p99:6us

hf3fs_fuse_main.1.log:[2025-03-12T10:15:28.982326782+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8299 mean:3636us min:1155us max:15688us p50:4141us p90:5634us p95:5955us p99:6489us
hf3fs_fuse_main.1.log:[2025-03-12T10:15:58.983104338+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8062 mean:3624us min:1189us max:16883us p50:4138us p90:5661us p95:6006us p99:6574us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.987233855+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7745 mean:3776us min:1066us max:16177us p50:4208us p90:5644us p95:5951us p99:6477us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.988463541+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8065 mean:3619us min:1190us max:15693us p50:4155us p90:5661us p95:6001us p99:6555us
hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.987235906+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7817 mean:3866us min:1038us max:16391us p50:4332us p90:5739us p95:6065us p99:6658us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.978109851+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:2749 mean:3488us min:1172us max:16699us p50:4080us p90:5670us p95:5996us p99:6605us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.985832600+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7808 mean:3863us min:1182us max:16385us p50:4317us p90:5753us p95:6079us p99:6629us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:26.983411905+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8391 mean:3586us min:1093us max:16562us p50:4135us p90:5679us p95:6038us p99:6567us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:57.983947149+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8046 mean:3751us min:1009us max:15743us p50:4280us p90:5666us p95:5971us p99:6505us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.990729069+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:9864 mean:100466us min:1224us max:168697us p50:133777us p90:146405us p95:149303us p99:154436us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.990681823+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10063 mean:98475us min:1252us max:167693us p50:133959us p90:146847us p95:149949us p99:156314us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.990822116+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10110 mean:98015us min:1204us max:167653us p50:132200us p90:146570us p95:151570us p99:159883us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.982803227+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:6651 mean:101551us min:1236us max:165213us p50:132984us p90:144065us p95:147014us p99:153712us
hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.987821095+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10026 mean:98779us min:1244us max:178294us p50:135884us p90:147044us p95:150563us p99:160114us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.990850994+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:9859 mean:97279us min:1177us max:170968us p50:132669us p90:146564us p95:150251us p99:159332us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.992533541+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10122 mean:97881us min:1221us max:167460us p50:131931us p90:145861us p95:148605us p99:155137us
hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.976552843+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:230 mean:121631us min:24422us max:142972us p50:130258us p90:139692us p95:140628us p99:142629us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.990988686+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10112 mean:98030us min:1227us max:181985us p50:133405us p90:145937us p95:148712us p99:155955us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.991496214+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10057 mean:98559us min:1219us max:171107us p50:131647us p90:148031us p95:151932us p99:158934us
hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.990544803+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10086 mean:98275us min:1249us max:178706us p50:134501us p90:145106us p95:148275us p99:156460us
hf3fs_fuse_main.log:[2025-03-12T10:17:59.989021673+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8300 mean:3516us min:1189us max:15827us p50:4023us p90:5679us p95:5982us p99:6659us
hf3fs_fuse_main.log:[2025-03-12T10:18:30.979916431+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:3983 mean:3692us min:1190us max:7555us p50:4271us p90:5759us p95:6059us p99:6553us

hf3fs_fuse_main.1.log:[2025-03-12T10:15:57.980095224+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8072 mean:3744us min:1191us max:16884us p50:4210us p90:5699us p95:6032us p99:6583us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.979868966+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8127 mean:3717us min:1068us max:16179us p50:4183us p90:5642us p95:5950us p99:6480us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.980175901+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8061 mean:3622us min:1191us max:15694us p50:4157us p90:5662us p95:6003us p99:6556us
hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.979907753+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7822 mean:3865us min:1038us max:16392us p50:4332us p90:5740us p95:6066us p99:6659us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.974886674+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:2748 mean:3489us min:1173us max:16701us p50:4081us p90:5671us p95:5998us p99:6606us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.977557071+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7807 mean:3865us min:1184us max:16386us p50:4318us p90:5754us p95:6079us p99:6631us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:25.977428090+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8165 mean:3568us min:1095us max:16564us p50:4131us p90:5695us p95:6046us p99:6569us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:55.979838602+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7479 mean:3910us min:1010us max:15745us p50:4353us p90:5684us p95:5985us p99:6529us
hf3fs_fuse_main.2.log:[2025-03-12T10:15:26.980866304+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8700 mean:3465us min:1157us max:15689us p50:4024us p90:5568us p95:5919us p99:6468us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.981236169+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:9864 mean:100469us min:1225us max:168698us p50:133789us p90:146419us p95:149304us p99:154438us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.982151783+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10063 mean:98476us min:1254us max:167695us p50:133957us p90:146849us p95:149950us p99:156316us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.980943833+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10110 mean:98019us min:1205us max:167655us p50:132188us p90:146569us p95:151571us p99:159885us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.976965384+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:6653 mean:101562us min:1238us max:165214us p50:132999us p90:144067us p95:147020us p99:153712us
hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.978112288+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10025 mean:98769us min:1245us max:178296us p50:135877us p90:147052us p95:150565us p99:160120us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.982755658+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:9860 mean:97285us min:1179us max:170970us p50:132676us p90:146565us p95:150252us p99:159333us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.983712357+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10122 mean:97881us min:1223us max:167462us p50:131940us p90:145849us p95:148606us p99:155138us
hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.974759620+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:230 mean:121633us min:24434us max:142973us p50:130260us p90:139694us p95:140630us p99:142631us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.982679122+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10111 mean:98027us min:1229us max:181987us p50:133411us p90:145939us p95:148714us p99:155957us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.982953347+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10056 mean:98558us min:1220us max:171109us p50:131662us p90:148033us p95:151934us p99:158936us
hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.982357297+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10086 mean:98282us min:1250us max:178708us p50:134498us p90:145108us p95:148277us p99:156462us
hf3fs_fuse_main.log:[2025-03-12T10:17:59.980325244+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8299 mean:3517us min:1190us max:15828us p50:4024us p90:5680us p95:5983us p99:6661us
hf3fs_fuse_main.log:[2025-03-12T10:18:30.976200164+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:3985 mean:3694us min:1192us max:7557us p50:4273us p90:5761us p95:6061us p99:6554us

hf3fs_fuse_main.1.log:[2025-03-12T10:15:28.977051715+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8300 mean:3642us min:1161us max:15693us p50:4146us p90:5640us p95:5960us p99:6494us
hf3fs_fuse_main.1.log:[2025-03-12T10:15:58.977497340+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8062 mean:3630us min:1195us max:16889us p50:4144us p90:5666us p95:6012us p99:6579us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.982964651+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7745 mean:3782us min:1069us max:16184us p50:4215us p90:5650us p95:5957us p99:6483us
hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.983389633+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8062 mean:3626us min:1196us max:15698us p50:4162us p90:5667us p95:6007us p99:6561us
hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.983066733+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7821 mean:3871us min:1040us max:16396us p50:4337us p90:5745us p95:6071us p99:6664us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.976188101+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:2749 mean:3494us min:1177us max:16713us p50:4087us p90:5677us p95:6003us p99:6611us
hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.980772958+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7807 mean:3870us min:1188us max:16391us p50:4323us p90:5759us p95:6084us p99:6636us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:26.977682224+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8391 mean:3593us min:1096us max:16571us p50:4143us p90:5687us p95:6045us p99:6574us
hf3fs_fuse_main.2.log:[2025-03-12T10:14:57.977958524+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8045 mean:3757us min:1012us max:15749us p50:4286us p90:5672us p95:5977us p99:6510us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.984806648+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:9863 mean:100469us min:1230us max:168705us p50:133790us p90:146418us p95:149310us p99:154443us
hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.985569232+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10064 mean:98485us min:1258us max:167699us p50:133964us p90:146853us p95:149954us p99:156320us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.984651268+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10110 mean:98023us min:1212us max:167661us p50:132200us p90:146575us p95:151576us p99:159890us
hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.979100253+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:6652 mean:101563us min:1243us max:165218us p50:132992us p90:144071us p95:147020us p99:153718us
hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.981861331+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10025 mean:98778us min:1251us max:178302us p50:135890us p90:147052us p95:150572us p99:160127us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.986015712+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:9860 mean:97290us min:1183us max:170974us p50:132682us p90:146570us p95:150257us p99:159337us
hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.987218407+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10122 mean:97887us min:1228us max:167470us p50:131937us p90:145861us p95:148611us p99:155143us
hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.975395145+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:230 mean:121642us min:24474us max:142978us p50:130265us p90:139703us p95:140636us p99:142637us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.985762524+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10111 mean:98032us min:1233us max:181994us p50:133415us p90:145944us p95:148719us p99:155963us
hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.986025216+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10057 mean:98567us min:1225us max:171114us p50:131663us p90:148037us p95:151938us p99:158941us
hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.985910918+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10086 mean:98285us min:1255us max:178713us p50:134505us p90:145113us p95:148283us p99:156467us
hf3fs_fuse_main.log:[2025-03-12T10:17:59.983575822+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8299 mean:3522us min:1195us max:15832us p50:4029us p90:5685us p95:5988us p99:6665us
hf3fs_fuse_main.log:[2025-03-12T10:18:30.977594859+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:3984 mean:3699us min:1196us max:7561us p50:4277us p90:5765us p95:6065us p99:6559us
Originally created by @222luhang on GitHub (Mar 12, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/160 在默认配置下进行的fio测试(depth修改32-->1) 读 ``` fio -ioengine=libaio -directory=/mnt/fio_test -name=test -size=1G -rw=read -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=300 -iodepth=1 -direct=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=6556KiB/s][r=1639 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=736789: Wed Mar 12 10:25:26 2025 read: IOPS=1475, BW=5902KiB/s (6044kB/s)(1729MiB/300001msec) slat (nsec): min=4722, max=163445, avg=6706.66, stdev=882.60 clat (usec): min=399, max=15810, avg=669.57, stdev=109.33 lat (usec): min=407, max=15816, avg=676.53, stdev=109.36 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 545], 10.00th=[ 553], 20.00th=[ 562], | 30.00th=[ 578], 40.00th=[ 586], 50.00th=[ 660], 60.00th=[ 725], | 70.00th=[ 750], 80.00th=[ 783], 90.00th=[ 816], 95.00th=[ 832], | 99.00th=[ 865], 99.50th=[ 873], 99.90th=[ 922], 99.95th=[ 955], | 99.99th=[ 1074] bw ( KiB/s): min= 4968, max= 7088, per=100.00%, avg=5902.88, stdev=445.03, samples=599 iops : min= 1242, max= 1772, avg=1475.72, stdev=111.26, samples=599 lat (usec) : 500=0.05%, 750=71.24%, 1000=28.68% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=0.54%, sys=1.67%, ctx=442648, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=442646,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=5902KiB/s (6044kB/s), 5902KiB/s-5902KiB/s (6044kB/s-6044kB/s), io=1729MiB (1813MB), run=300001-300001msec ``` 写 ``` fio -ioengine=libaio -directory=/mnt/fio_test -name=test -size=1G -rw=write -bs=4k -numjobs=1 --group_reporting -time_based=1 -runtime=300 -iodepth=1 -direct=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=1388KiB/s][w=347 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=736706: Wed Mar 12 10:18:15 2025 write: IOPS=264, BW=1057KiB/s (1082kB/s)(310MiB/300006msec); 0 zone resets slat (usec): min=2, max=259, avg= 6.17, stdev= 1.08 clat (usec): min=1059, max=16990, avg=3777.01, stdev=1759.73 lat (usec): min=1066, max=16996, avg=3783.48, stdev=1759.79 clat percentiles (usec): | 1.00th=[ 1287], 5.00th=[ 1303], 10.00th=[ 1303], 20.00th=[ 1319], | 30.00th=[ 1549], 40.00th=[ 3982], 50.00th=[ 4293], 60.00th=[ 4555], | 70.00th=[ 4883], 80.00th=[ 5276], 90.00th=[ 5800], 95.00th=[ 6128], | 99.00th=[ 6652], 99.50th=[ 6849], 99.90th=[ 7373], 99.95th=[ 8291], | 99.99th=[15926] bw ( KiB/s): min= 696, max= 3024, per=100.00%, avg=1057.50, stdev=507.82, samples=599 iops : min= 174, max= 756, avg=264.37, stdev=126.95, samples=599 lat (msec) : 2=30.25%, 4=10.00%, 10=59.72%, 20=0.03% cpu : usr=0.12%, sys=0.27%, ctx=79267, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,79265,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=1057KiB/s (1082kB/s), 1057KiB/s-1057KiB/s (1082kB/s-1082kB/s), io=310MiB (325MB), run=300006-300006msec ``` 客户端日志中的时延统计，耗时最多在submit ``` hf3fs_fuse_main.1.log:[2025-03-12T10:15:57.977300439+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8072 mean:4us min:1us max:18us p50:4us p90:5us p95:5us p99:6us hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.977022018+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8127 mean:4us min:1us max:18us p50:4us p90:5us p95:5us p99:7us hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.977352049+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8060 mean:4us min:3us max:32us p50:4us p90:5us p95:5us p99:8us hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.977142518+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7824 mean:4us min:1us max:37us p50:4us p90:5us p95:5us p99:6us hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.973805710+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:2749 mean:5us min:2us max:32us p50:4us p90:6us p95:8us p99:9us hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.974500515+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7807 mean:5us min:2us max:29us p50:4us p90:7us p95:8us p99:9us hf3fs_fuse_main.2.log:[2025-03-12T10:14:25.974563626+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8164 mean:5us min:1us max:40us p50:4us p90:7us p95:8us p99:9us hf3fs_fuse_main.2.log:[2025-03-12T10:14:55.977177490+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:7479 mean:4us min:2us max:69us p50:4us p90:5us p95:6us p99:8us hf3fs_fuse_main.2.log:[2025-03-12T10:15:26.977794989+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8700 mean:4us min:3us max:19us p50:4us p90:5us p95:5us p99:6us hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.978103726+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:9864 mean:5us min:1us max:36us p50:5us p90:6us p95:6us p99:7us hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.978654351+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10063 mean:5us min:3us max:51us p50:5us p90:6us p95:6us p99:8us hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.977743566+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10110 mean:5us min:3us max:41us p50:5us p90:6us p95:6us p99:7us hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.975045407+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:6622 mean:5us min:1us max:40us p50:5us p90:6us p95:6us p99:8us hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.974779973+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10026 mean:5us min:3us max:41us p50:5us p90:6us p95:6us p99:8us hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.979509807+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:9859 mean:5us min:1us max:97us p50:5us p90:6us p95:6us p99:8us hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.980144439+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10122 mean:5us min:1us max:44us p50:5us p90:6us p95:6us p99:7us hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.974228808+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:262 mean:7us min:3us max:61us p50:5us p90:12us p95:14us p99:39us hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.979397655+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10110 mean:5us min:2us max:31us p50:5us p90:6us p95:7us p99:9us hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.979719352+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10056 mean:5us min:3us max:48us p50:5us p90:6us p95:7us p99:9us hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.979134929+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:10086 mean:5us min:1us max:75us p50:5us p90:6us p95:6us p99:8us hf3fs_fuse_main.log:[2025-03-12T10:17:59.977448568+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:8299 mean:4us min:1us max:23us p50:4us p90:5us p95:5us p99:7us hf3fs_fuse_main.log:[2025-03-12T10:18:30.974932857+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.prepare.write: count:3984 mean:4us min:2us max:40us p50:4us p90:5us p95:5us p99:6us hf3fs_fuse_main.1.log:[2025-03-12T10:15:28.982326782+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8299 mean:3636us min:1155us max:15688us p50:4141us p90:5634us p95:5955us p99:6489us hf3fs_fuse_main.1.log:[2025-03-12T10:15:58.983104338+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8062 mean:3624us min:1189us max:16883us p50:4138us p90:5661us p95:6006us p99:6574us hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.987233855+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7745 mean:3776us min:1066us max:16177us p50:4208us p90:5644us p95:5951us p99:6477us hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.988463541+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8065 mean:3619us min:1190us max:15693us p50:4155us p90:5661us p95:6001us p99:6555us hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.987235906+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7817 mean:3866us min:1038us max:16391us p50:4332us p90:5739us p95:6065us p99:6658us hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.978109851+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:2749 mean:3488us min:1172us max:16699us p50:4080us p90:5670us p95:5996us p99:6605us hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.985832600+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:7808 mean:3863us min:1182us max:16385us p50:4317us p90:5753us p95:6079us p99:6629us hf3fs_fuse_main.2.log:[2025-03-12T10:14:26.983411905+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8391 mean:3586us min:1093us max:16562us p50:4135us p90:5679us p95:6038us p99:6567us hf3fs_fuse_main.2.log:[2025-03-12T10:14:57.983947149+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8046 mean:3751us min:1009us max:15743us p50:4280us p90:5666us p95:5971us p99:6505us hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.990729069+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:9864 mean:100466us min:1224us max:168697us p50:133777us p90:146405us p95:149303us p99:154436us hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.990681823+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10063 mean:98475us min:1252us max:167693us p50:133959us p90:146847us p95:149949us p99:156314us hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.990822116+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10110 mean:98015us min:1204us max:167653us p50:132200us p90:146570us p95:151570us p99:159883us hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.982803227+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:6651 mean:101551us min:1236us max:165213us p50:132984us p90:144065us p95:147014us p99:153712us hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.987821095+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10026 mean:98779us min:1244us max:178294us p50:135884us p90:147044us p95:150563us p99:160114us hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.990850994+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:9859 mean:97279us min:1177us max:170968us p50:132669us p90:146564us p95:150251us p99:159332us hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.992533541+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10122 mean:97881us min:1221us max:167460us p50:131931us p90:145861us p95:148605us p99:155137us hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.976552843+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:230 mean:121631us min:24422us max:142972us p50:130258us p90:139692us p95:140628us p99:142629us hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.990988686+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10112 mean:98030us min:1227us max:181985us p50:133405us p90:145937us p95:148712us p99:155955us hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.991496214+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10057 mean:98559us min:1219us max:171107us p50:131647us p90:148031us p95:151932us p99:158934us hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.990544803+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:10086 mean:98275us min:1249us max:178706us p50:134501us p90:145106us p95:148275us p99:156460us hf3fs_fuse_main.log:[2025-03-12T10:17:59.989021673+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:8300 mean:3516us min:1189us max:15827us p50:4023us p90:5679us p95:5982us p99:6659us hf3fs_fuse_main.log:[2025-03-12T10:18:30.979916431+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.submit.write: count:3983 mean:3692us min:1190us max:7555us p50:4271us p90:5759us p95:6059us p99:6553us hf3fs_fuse_main.1.log:[2025-03-12T10:15:57.980095224+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8072 mean:3744us min:1191us max:16884us p50:4210us p90:5699us p95:6032us p99:6583us hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.979868966+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8127 mean:3717us min:1068us max:16179us p50:4183us p90:5642us p95:5950us p99:6480us hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.980175901+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8061 mean:3622us min:1191us max:15694us p50:4157us p90:5662us p95:6003us p99:6556us hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.979907753+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7822 mean:3865us min:1038us max:16392us p50:4332us p90:5740us p95:6066us p99:6659us hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.974886674+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:2748 mean:3489us min:1173us max:16701us p50:4081us p90:5671us p95:5998us p99:6606us hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.977557071+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7807 mean:3865us min:1184us max:16386us p50:4318us p90:5754us p95:6079us p99:6631us hf3fs_fuse_main.2.log:[2025-03-12T10:14:25.977428090+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8165 mean:3568us min:1095us max:16564us p50:4131us p90:5695us p95:6046us p99:6569us hf3fs_fuse_main.2.log:[2025-03-12T10:14:55.979838602+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:7479 mean:3910us min:1010us max:15745us p50:4353us p90:5684us p95:5985us p99:6529us hf3fs_fuse_main.2.log:[2025-03-12T10:15:26.980866304+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8700 mean:3465us min:1157us max:15689us p50:4024us p90:5568us p95:5919us p99:6468us hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.981236169+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:9864 mean:100469us min:1225us max:168698us p50:133789us p90:146419us p95:149304us p99:154438us hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.982151783+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10063 mean:98476us min:1254us max:167695us p50:133957us p90:146849us p95:149950us p99:156316us hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.980943833+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10110 mean:98019us min:1205us max:167655us p50:132188us p90:146569us p95:151571us p99:159885us hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.976965384+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:6653 mean:101562us min:1238us max:165214us p50:132999us p90:144067us p95:147020us p99:153712us hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.978112288+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10025 mean:98769us min:1245us max:178296us p50:135877us p90:147052us p95:150565us p99:160120us hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.982755658+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:9860 mean:97285us min:1179us max:170970us p50:132676us p90:146565us p95:150252us p99:159333us hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.983712357+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10122 mean:97881us min:1223us max:167462us p50:131940us p90:145849us p95:148606us p99:155138us hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.974759620+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:230 mean:121633us min:24434us max:142973us p50:130260us p90:139694us p95:140630us p99:142631us hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.982679122+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10111 mean:98027us min:1229us max:181987us p50:133411us p90:145939us p95:148714us p99:155957us hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.982953347+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10056 mean:98558us min:1220us max:171109us p50:131662us p90:148033us p95:151934us p99:158936us hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.982357297+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:10086 mean:98282us min:1250us max:178708us p50:134498us p90:145108us p95:148277us p99:156462us hf3fs_fuse_main.log:[2025-03-12T10:17:59.980325244+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:8299 mean:3517us min:1190us max:15828us p50:4024us p90:5680us p95:5983us p99:6661us hf3fs_fuse_main.log:[2025-03-12T10:18:30.976200164+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.complete.write: count:3985 mean:3694us min:1192us max:7557us p50:4273us p90:5761us p95:6061us p99:6554us hf3fs_fuse_main.1.log:[2025-03-12T10:15:28.977051715+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8300 mean:3642us min:1161us max:15693us p50:4146us p90:5640us p95:5960us p99:6494us hf3fs_fuse_main.1.log:[2025-03-12T10:15:58.977497340+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8062 mean:3630us min:1195us max:16889us p50:4144us p90:5666us p95:6012us p99:6579us hf3fs_fuse_main.1.log:[2025-03-12T10:16:28.982964651+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7745 mean:3782us min:1069us max:16184us p50:4215us p90:5650us p95:5957us p99:6483us hf3fs_fuse_main.1.log:[2025-03-12T10:16:58.983389633+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8062 mean:3626us min:1196us max:15698us p50:4162us p90:5667us p95:6007us p99:6561us hf3fs_fuse_main.1.log:[2025-03-12T10:17:29.983066733+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7821 mean:3871us min:1040us max:16396us p50:4337us p90:5745us p95:6071us p99:6664us hf3fs_fuse_main.2.log:[2025-03-12T10:13:24.976188101+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:2749 mean:3494us min:1177us max:16713us p50:4087us p90:5677us p95:6003us p99:6611us hf3fs_fuse_main.2.log:[2025-03-12T10:13:55.980772958+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:7807 mean:3870us min:1188us max:16391us p50:4323us p90:5759us p95:6084us p99:6636us hf3fs_fuse_main.2.log:[2025-03-12T10:14:26.977682224+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8391 mean:3593us min:1096us max:16571us p50:4143us p90:5687us p95:6045us p99:6574us hf3fs_fuse_main.2.log:[2025-03-12T10:14:57.977958524+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8045 mean:3757us min:1012us max:15749us p50:4286us p90:5672us p95:5977us p99:6510us hf3fs_fuse_main.3.log:[2025-03-12T10:11:20.984806648+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:9863 mean:100469us min:1230us max:168705us p50:133790us p90:146418us p95:149310us p99:154443us hf3fs_fuse_main.3.log:[2025-03-12T10:11:51.985569232+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10064 mean:98485us min:1258us max:167699us p50:133964us p90:146853us p95:149954us p99:156320us hf3fs_fuse_main.3.log:[2025-03-12T10:12:22.984651268+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10110 mean:98023us min:1212us max:167661us p50:132200us p90:146575us p95:151576us p99:159890us hf3fs_fuse_main.3.log:[2025-03-12T10:12:53.979100253+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:6652 mean:101563us min:1243us max:165218us p50:132992us p90:144071us p95:147020us p99:153718us hf3fs_fuse_main.4.log:[2025-03-12T10:09:48.981861331+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10025 mean:98778us min:1251us max:178302us p50:135890us p90:147052us p95:150572us p99:160127us hf3fs_fuse_main.4.log:[2025-03-12T10:10:18.986015712+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:9860 mean:97290us min:1183us max:170974us p50:132682us p90:146570us p95:150257us p99:159337us hf3fs_fuse_main.4.log:[2025-03-12T10:10:49.987218407+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10122 mean:97887us min:1228us max:167470us p50:131937us p90:145861us p95:148611us p99:155143us hf3fs_fuse_main.5.log:[2025-03-12T10:07:44.975395145+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:230 mean:121642us min:24474us max:142978us p50:130265us p90:139703us p95:140636us p99:142637us hf3fs_fuse_main.5.log:[2025-03-12T10:08:15.985762524+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10111 mean:98032us min:1233us max:181994us p50:133415us p90:145944us p95:148719us p99:155963us hf3fs_fuse_main.5.log:[2025-03-12T10:08:46.986025216+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10057 mean:98567us min:1225us max:171114us p50:131663us p90:148037us p95:151938us p99:158941us hf3fs_fuse_main.5.log:[2025-03-12T10:09:17.985910918+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:10086 mean:98285us min:1255us max:178713us p50:134505us p90:145113us p95:148283us p99:156467us hf3fs_fuse_main.log:[2025-03-12T10:17:59.983575822+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:8299 mean:3522us min:1195us max:15832us p50:4029us p90:5685us p95:5988us p99:6665us hf3fs_fuse_main.log:[2025-03-12T10:18:30.977594859+00:00 Collector:736010 Recorder.cc:242 DEBUG] Last 30s fuse.piov.overall.write: count:3984 mean:3699us min:1196us max:7561us p50:4277us p90:5765us p95:6065us p99:6559us ```
bsakaguchi commented 2026-10-26 09:45:02 +00:00
Author
Owner
Copy link

@bigbrotherwei commented on GitHub (Mar 12, 2025):

测时延iodepth应该是1吧，看看iodepth=1的时延多少

<!-- gh-comment-id:2717337442 --> @bigbrotherwei commented on GitHub (Mar 12, 2025): 测时延iodepth应该是1吧，看看iodepth=1的时延多少
bsakaguchi commented 2026-10-26 09:45:02 +00:00
Author
Owner
Copy link

@SF-Zhou commented on GitHub (Mar 13, 2025):

3FS is not optimized for small writes. Try writing in large chunks.

<!-- gh-comment-id:2719466488 --> @SF-Zhou commented on GitHub (Mar 13, 2025): 3FS is not optimized for small writes. Try writing in large chunks.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#96
No description provided.