kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

xxx #55

New issue

Closed

opened 2026-10-29 21:08:31 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:08:31 +00:00

Owner

Originally created by @Huixxi on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/105

Originally created by @Huixxi on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/105 ![Image](https://github.com/user-attachments/assets/91f15c8d-0638-41c9-a992-68c7d0acbe17)

bsakaguchi closed this issue

2026-10-29 21:08:31 +00:00

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@Huixxi commented on GitHub (Feb 14, 2025):

暂时解决了，但是无法获取IB设备。

@Huixxi commented on GitHub (Feb 14, 2025): 暂时解决了，但是无法获取IB设备。

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@alogfans commented on GitHub (Feb 14, 2025):

这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。

@alogfans commented on GitHub (Feb 14, 2025): 这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@Huixxi commented on GitHub (Feb 15, 2025):

好的谢谢，但是--device_name应该如何获取到呢？而且后面我用的--auto_discovery虽然成功了，但是还是提示No IB
devices found
./transfer_engine_bench --mode=target
--metadata_server=33.xx.xx.64:7004
--local_server_name=33.xx.xx.67:7004
--auto_discovery
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0214 16:46:52.014591 171152 topology.cpp:53] No IB devices found
I0214 16:46:53.058771 171152 transfer_engine_bench.cpp:346] numa node num: 2

On Fri, Feb 14, 2025 at 8:17 PM Feng Ren @.***> wrote:

这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。

—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNPWGOE6XO4GSG42FVL2PXND5AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZGE4DKNBZGQ
.
You are receiving this because you modified the open/close state.Message
ID: @.***>
[image: alogfans]alogfans left a comment (kvcache-ai/Mooncake#105)
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494

这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。

—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNPWGOE6XO4GSG42FVL2PXND5AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZGE4DKNBZGQ
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

@Huixxi commented on GitHub (Feb 15, 2025): 好的谢谢，但是--device_name应该如何获取到呢？而且后面我用的--auto_discovery虽然成功了，但是还是提示No IB devices found ./transfer_engine_bench --mode=target \ --metadata_server=33.xx.xx.64:7004 \ --local_server_name=33.xx.xx.67:7004 \ --auto_discovery WARNING: Logging before InitGoogleLogging() is written to STDERR W0214 16:46:52.014591 171152 topology.cpp:53] No IB devices found I0214 16:46:53.058771 171152 transfer_engine_bench.cpp:346] numa node num: 2 On Fri, Feb 14, 2025 at 8:17 PM Feng Ren ***@***.***> wrote: > 这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNPWGOE6XO4GSG42FVL2PXND5AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZGE4DKNBZGQ> > . > You are receiving this because you modified the open/close state.Message > ID: ***@***.***> > [image: alogfans]*alogfans* left a comment (kvcache-ai/Mooncake#105) > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494> > > 这是因为你指定了 --protocol=tcp 模式，所以不会获取 IB 设备。 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2659185494>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNPWGOE6XO4GSG42FVL2PXND5AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZGE4DKNBZGQ> > . > You are receiving this because you modified the open/close state.Message > ID: ***@***.***> >

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@doujiang24 commented on GitHub (Feb 15, 2025):

@Huixxi 你用 ibv_devices 能看到 rdma 网卡么？正常情况下的输出示例：

    device                 node GUID
    ------              ----------------
    mlx5_bond_0         08c0eb0300da0fce
    mlx5_bond_1         08c0eb0300da0fc2

@doujiang24 commented on GitHub (Feb 15, 2025): @Huixxi 你用 ibv_devices 能看到 rdma 网卡么？正常情况下的输出示例： ``` device node GUID ------ ---------------- mlx5_bond_0 08c0eb0300da0fce mlx5_bond_1 08c0eb0300da0fc2 ```

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@Huixxi commented on GitHub (Feb 16, 2025):

我使用
$ ibv_devices
device node GUID
------ ----------------
是看不到的，但是用下面的命令又能列出来：
$ ls /sys/class/infiniband/
mlx5_0 mlx5_10 mlx5_12 mlx5_14 mlx5_16 mlx5_18 mlx5_2 mlx5_21
mlx5_23 mlx5_25 mlx5_27 mlx5_29 mlx5_30 mlx5_32 mlx5_34 mlx5_36
mlx5_38 mlx5_4 mlx5_41 mlx5_6 mlx5_8 mlx5_bond_0 mlx5_bond_2
mlx5_1 mlx5_11 mlx5_13 mlx5_15 mlx5_17 mlx5_19 mlx5_20 mlx5_22
mlx5_24 mlx5_26 mlx5_28 mlx5_3 mlx5_31 mlx5_33 mlx5_35 mlx5_37
mlx5_39 mlx5_40 mlx5_5 mlx5_7 mlx5_9 mlx5_bond_1 mlx5_bond_3

还有下面的命令：
$ rdma link | grep -v DISABLE
link mlx5_bond_0/9 state ACTIVE physical_state LINK_UP
link mlx5_bond_0/64 state ACTIVE physical_state LINK_UP
link mlx5_bond_1/7 state ACTIVE physical_state LINK_UP
link mlx5_bond_1/64 state ACTIVE physical_state LINK_UP
link mlx5_bond_2/3 state ACTIVE physical_state LINK_UP
link mlx5_bond_2/4 state ACTIVE physical_state LINK_UP
link mlx5_bond_2/64 state ACTIVE physical_state LINK_UP
link mlx5_bond_3/3 state ACTIVE physical_state LINK_UP
link mlx5_bond_3/64 state ACTIVE physical_state LINK_UP
link mlx5_5/1 state ACTIVE physical_state LINK_UP
link mlx5_8/1 state ACTIVE physical_state LINK_UP
link mlx5_25/1 state ACTIVE physical_state LINK_UP
link mlx5_27/1 state ACTIVE physical_state LINK_UP netdev eth0
link mlx5_39/1 state ACTIVE physical_state LINK_UP

On Sat, Feb 15, 2025 at 9:39 PM doujiang24 @.***> wrote:

@Huixxi https://github.com/Huixxi 你用 ibv_devices 能看到 rdma
网卡么？正常情况下的输出示例：
device                 node GUID
------              ----------------
mlx5_bond_0         08c0eb0300da0fce
mlx5_bond_1         08c0eb0300da0fc2
—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNINBW2CCQ7FMPLTRXL2P47P3AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQHEZDSNRWGU
.
You are receiving this because you were mentioned.Message ID:
@.***>
[image: doujiang24]doujiang24 left a comment (kvcache-ai/Mooncake#105)
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665

@Huixxi https://github.com/Huixxi 你用 ibv_devices 能看到 rdma
网卡么？正常情况下的输出示例：
device                 node GUID
------              ----------------
mlx5_bond_0         08c0eb0300da0fce
mlx5_bond_1         08c0eb0300da0fc2
—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNINBW2CCQ7FMPLTRXL2P47P3AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQHEZDSNRWGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

@Huixxi commented on GitHub (Feb 16, 2025): 我使用 $ ibv_devices device node GUID ------ ---------------- 是看不到的，但是用下面的命令又能列出来： $ ls /sys/class/infiniband/ mlx5_0 mlx5_10 mlx5_12 mlx5_14 mlx5_16 mlx5_18 mlx5_2 mlx5_21 mlx5_23 mlx5_25 mlx5_27 mlx5_29 mlx5_30 mlx5_32 mlx5_34 mlx5_36 mlx5_38 mlx5_4 mlx5_41 mlx5_6 mlx5_8 mlx5_bond_0 mlx5_bond_2 mlx5_1 mlx5_11 mlx5_13 mlx5_15 mlx5_17 mlx5_19 mlx5_20 mlx5_22 mlx5_24 mlx5_26 mlx5_28 mlx5_3 mlx5_31 mlx5_33 mlx5_35 mlx5_37 mlx5_39 mlx5_40 mlx5_5 mlx5_7 mlx5_9 mlx5_bond_1 mlx5_bond_3 还有下面的命令： $ rdma link | grep -v DISABLE link mlx5_bond_0/9 state ACTIVE physical_state LINK_UP link mlx5_bond_0/64 state ACTIVE physical_state LINK_UP link mlx5_bond_1/7 state ACTIVE physical_state LINK_UP link mlx5_bond_1/64 state ACTIVE physical_state LINK_UP link mlx5_bond_2/3 state ACTIVE physical_state LINK_UP link mlx5_bond_2/4 state ACTIVE physical_state LINK_UP link mlx5_bond_2/64 state ACTIVE physical_state LINK_UP link mlx5_bond_3/3 state ACTIVE physical_state LINK_UP link mlx5_bond_3/64 state ACTIVE physical_state LINK_UP link mlx5_5/1 state ACTIVE physical_state LINK_UP link mlx5_8/1 state ACTIVE physical_state LINK_UP link mlx5_25/1 state ACTIVE physical_state LINK_UP link mlx5_27/1 state ACTIVE physical_state LINK_UP netdev eth0 link mlx5_39/1 state ACTIVE physical_state LINK_UP On Sat, Feb 15, 2025 at 9:39 PM doujiang24 ***@***.***> wrote: > @Huixxi <https://github.com/Huixxi> 你用 ibv_devices 能看到 rdma > 网卡么？正常情况下的输出示例： > > device node GUID > ------ ---------------- > mlx5_bond_0 08c0eb0300da0fce > mlx5_bond_1 08c0eb0300da0fc2 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNINBW2CCQ7FMPLTRXL2P47P3AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQHEZDSNRWGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > [image: doujiang24]*doujiang24* left a comment (kvcache-ai/Mooncake#105) > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665> > > @Huixxi <https://github.com/Huixxi> 你用 ibv_devices 能看到 rdma > 网卡么？正常情况下的输出示例： > > device node GUID > ------ ---------------- > mlx5_bond_0 08c0eb0300da0fce > mlx5_bond_1 08c0eb0300da0fc2 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2660929665>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNINBW2CCQ7FMPLTRXL2P47P3AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQHEZDSNRWGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@doujiang24 commented on GitHub (Feb 17, 2025):

@Huixxi 这种可能是 ibverbs 的版本不对，可以找下供应商问问的

ibv_devices 能查到问题，应该就可以解决了

@doujiang24 commented on GitHub (Feb 17, 2025): @Huixxi 这种可能是 ibverbs 的版本不对，可以找下供应商问问的 ibv_devices 能查到问题，应该就可以解决了

bsakaguchi commented

2026-10-29 21:08:32 +00:00

Author

Owner

@Huixxi commented on GitHub (Feb 17, 2025):

好的谢谢大佬解答。

On Mon, Feb 17, 2025 at 10:24 AM doujiang24 @.***>
wrote:

@Huixxi https://github.com/Huixxi 这种可能是 ibverbs 的版本不对，可以找下供应商问问的

ibv_devices 能查到问题，应该就可以解决了

—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNNXUXMY5DGWN5LXD2L2QFB47AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHAYTAOBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>
[image: doujiang24]doujiang24 left a comment (kvcache-ai/Mooncake#105)
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898

@Huixxi https://github.com/Huixxi 这种可能是 ibverbs 的版本不对，可以找下供应商问问的

ibv_devices 能查到问题，应该就可以解决了

—
Reply to this email directly, view it on GitHub
https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFCYYNNXUXMY5DGWN5LXD2L2QFB47AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHAYTAOBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@Huixxi commented on GitHub (Feb 17, 2025): 好的谢谢大佬解答。 On Mon, Feb 17, 2025 at 10:24 AM doujiang24 ***@***.***> wrote: > @Huixxi <https://github.com/Huixxi> 这种可能是 ibverbs 的版本不对，可以找下供应商问问的 > > ibv_devices 能查到问题，应该就可以解决了 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNNXUXMY5DGWN5LXD2L2QFB47AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHAYTAOBZHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > [image: doujiang24]*doujiang24* left a comment (kvcache-ai/Mooncake#105) > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898> > > @Huixxi <https://github.com/Huixxi> 这种可能是 ibverbs 的版本不对，可以找下供应商问问的 > > ibv_devices 能查到问题，应该就可以解决了 > > — > Reply to this email directly, view it on GitHub > <https://github.com/kvcache-ai/Mooncake/issues/105#issuecomment-2661810898>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFCYYNNXUXMY5DGWN5LXD2L2QFB47AVCNFSM6AAAAABXEAEL3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRHAYTAOBZHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >