deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #239] [Ask For Help] metadata operations works but failed to write data into storage servers. #145

New issue
Open
opened 2026-10-26 09:45:56 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:45:56 +00:00
Owner
Copy link

Originally created by @SimonCqk on GitHub (Apr 13, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/239

I have deployed 3FS in a production Kubernetes environment (RoCE RDMA container network). Components like Mgmtd and Meta appear normal, and the list-nodes command shows proper results. After mounting FUSE, metadata operations like directory listing work correctly, but data writes echo hello > /xxxx/hello.txt fail with the following errors. RDMA network connectivity between FUSE clients and Storage nodes has been verified

list node output

Image

stderr

[2025-04-13T07:18:58.223072746+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops 0x7f1fbc792758 after #3 retries, elapsed time: 60001ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223156990+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] #1/1 Failed batchRead operation 0x7f1fbc7641f0 with id RequestId(0), routing target: ChainId(900100014)@ChainVersion(2)@RoutingInfoVersion(18):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000003-E0040000-00000000), user: Uid(0)/Gid(0), usercall: #1, status: StorageClient::NotAvailable(7005)
[2025-04-13T07:18:58.223172010+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] Finally collected 1/1 failed batchRead operations 0x7f1fbc77d768 issued by user Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223277021+00:00 hf3fs_fuse_main:235453 FuseOps.cc:1544 ERROR]   hf3fs_read error, ino 0x000000000003e004, size 4096, off 0, err -7005

stdout

Image

Originally created by @SimonCqk on GitHub (Apr 13, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/239 I have deployed 3FS in a production Kubernetes environment (RoCE RDMA container network). Components like Mgmtd and Meta appear normal, and the `list-nodes` command shows proper results. After mounting FUSE, metadata operations like directory listing work correctly, but data writes `echo hello > /xxxx/hello.txt` fail with the following errors. RDMA network connectivity between FUSE clients and Storage nodes has been verified ### list node output ![Image](https://github.com/user-attachments/assets/9748f64f-c403-4c58-8bd2-f26e46cc1f98) ### stderr ``` [2025-04-13T07:18:58.223072746+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops 0x7f1fbc792758 after #3 retries, elapsed time: 60001ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #1 [2025-04-13T07:18:58.223156990+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] #1/1 Failed batchRead operation 0x7f1fbc7641f0 with id RequestId(0), routing target: ChainId(900100014)@ChainVersion(2)@RoutingInfoVersion(18):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000003-E0040000-00000000), user: Uid(0)/Gid(0), usercall: #1, status: StorageClient::NotAvailable(7005) [2025-04-13T07:18:58.223172010+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] Finally collected 1/1 failed batchRead operations 0x7f1fbc77d768 issued by user Uid(0)/Gid(0), usercall: #1 [2025-04-13T07:18:58.223277021+00:00 hf3fs_fuse_main:235453 FuseOps.cc:1544 ERROR] hf3fs_read error, ino 0x000000000003e004, size 4096, off 0, err -7005 ``` ### stdout ![Image](https://github.com/user-attachments/assets/14528708-ef08-47aa-b569-96510cd33411)
bsakaguchi commented 2026-10-26 09:45:57 +00:00
Author
Owner
Copy link

@SimonCqk commented on GitHub (Apr 13, 2025):

New stderr from storage-server side, it seems that targets creation were not completed as expected?

Image

Image

<!-- gh-comment-id:2799845508 --> @SimonCqk commented on GitHub (Apr 13, 2025): New stderr from storage-server side, it seems that targets creation were not completed as expected? ![Image](https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703) ![Image](https://github.com/user-attachments/assets/724d8960-462b-498f-9c3a-2a3800890a2d)
bsakaguchi commented 2026-10-26 09:45:57 +00:00
Author
Owner
Copy link

@Navacore commented on GitHub (Apr 20, 2025):

I don't know what they're accusing me of, please stop sending messages.

El dom, 13 de abr. de 2025 3:48 a. m., Simon_CQK @.***>
escribió:

New stderr from storage-server side, it seems that targets creation were
not completed as expected?

image.png (view on web)
https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703

—
Reply to this email directly, view it on GitHub
https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>
SimonCqk left a comment (deepseek-ai/3FS#239)
https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508

New stderr from storage-server side, it seems that targets creation were
not completed as expected?

image.png (view on web)
https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703

—
Reply to this email directly, view it on GitHub
https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

<!-- gh-comment-id:2817383387 --> @Navacore commented on GitHub (Apr 20, 2025): I don't know what they're accusing me of, please stop sending messages. El dom, 13 de abr. de 2025 3:48 a. m., Simon_CQK ***@***.***> escribió: > New stderr from storage-server side, it seems that targets creation were > not completed as expected? > > image.png (view on web) > <https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703> > > — > Reply to this email directly, view it on GitHub > <https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> > *SimonCqk* left a comment (deepseek-ai/3FS#239) > <https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508> > > New stderr from storage-server side, it seems that targets creation were > not completed as expected? > > image.png (view on web) > <https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703> > > — > Reply to this email directly, view it on GitHub > <https://github.com/deepseek-ai/3FS/issues/239#issuecomment-2799845508>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#145
No description provided.