[GH-ISSUE #281 ] sglang with mooncake prefill node fail with &#39;Failed to register memory&#39; error #114

Author

Owner

@weibingo commented on GitHub (Apr 29, 2025):

run transfer_engine_bench also this problem

transfer_engine_bench --mode=target
--metadata_server=http://10.112.x.x:8080/metadata
--local_server_name=10.112.x.5:3654
--protocol=rdma
--operation=write
--auto_discovery

target能正常启起来。
然后启动init就会报错
transfer_engine_bench
--metadata_server=http://10.112.x.x:8080/metadata
--segment_id=10.112.x.5:3654
--local_server_name=10.112.x.4:3655
--protocol=rdma
--operation=write
--auto_discovery

报错信息：
I0429 08:04:34.819869 1834 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 17, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:59:5a
I0429 08:04:34.823685 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_1/
I0429 08:04:34.824589 1834 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 23, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:6c:72
I0429 08:04:34.828298 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_2/
I0429 08:04:34.829383 1834 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 3, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:6d:06
I0429 08:04:34.833137 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_3/
I0429 08:04:34.834007 1834 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:58:56
I0429 08:04:34.834899 1834 transfer_engine_bench.cpp:299] VRAM is used
E0429 08:04:35.036568 1834 rdma_context.cpp:201] Failed to register memory 0x7fe520000000: Bad address [14]
F0429 08:04:35.036597 1834 transfer_engine_bench.cpp:305] Assert failed: !rc

添加参数--use_vram=0，则可以，但此参数是在cpu上申请的地址，想测试不同节点间gpu通信速度。为何在gpu上registerLocalMemory就失败呢

@weibingo commented on GitHub (Apr 29, 2025): run transfer_engine_bench also this problem transfer_engine_bench --mode=target \ --metadata_server=http://10.112.x.x:8080/metadata \ --local_server_name=10.112.x.5:3654 \ --protocol=rdma \ --operation=write \ --auto_discovery target能正常启起来。然后启动init就会报错 transfer_engine_bench \ --metadata_server=http://10.112.x.x:8080/metadata \ --segment_id=10.112.x.5:3654 \ --local_server_name=10.112.x.4:3655 \ --protocol=rdma \ --operation=write \ --auto_discovery 报错信息： I0429 08:04:34.819869 1834 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 17, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:59:5a I0429 08:04:34.823685 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_1/ I0429 08:04:34.824589 1834 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 23, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:6c:72 I0429 08:04:34.828298 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_2/ I0429 08:04:34.829383 1834 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 3, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:6d:06 I0429 08:04:34.833137 1834 rdma_context.cpp:426] Find best gid index: 0 on mlx5_3/ I0429 08:04:34.834007 1834 rdma_context.cpp:125] RDMA device: mlx5_3, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:96:58:56 I0429 08:04:34.834899 1834 transfer_engine_bench.cpp:299] VRAM is used E0429 08:04:35.036568 1834 rdma_context.cpp:201] Failed to register memory 0x7fe520000000: Bad address [14] F0429 08:04:35.036597 1834 transfer_engine_bench.cpp:305] Assert failed: !rc 添加参数--use_vram=0，则可以，但此参数是在cpu上申请的地址，想测试不同节点间gpu通信速度。为何在gpu上registerLocalMemory就失败呢

bsakaguchi commented

Author

Owner

@stmatengss commented on GitHub (May 1, 2025):

Before running benchmark, please check GDR is support or not. Could you please provide the hardware confifurations？

@stmatengss commented on GitHub (May 1, 2025): Before running benchmark, please check GDR is support or not. Could you please provide the hardware confifurations？

bsakaguchi commented

Author

Owner

@weibingo commented on GitHub (May 6, 2025):

Before running benchmark, please check GDR is support or not. Could you please provide the hardware confifurations？

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (May 6, 2025):

@weibingo could you please provide the result of the command: lsmod | grep nv.

@doujiang24 commented on GitHub (May 6, 2025): @weibingo could you please provide the result of the command: `lsmod | grep nv`.

bsakaguchi commented

Author

Owner

@bliu3650 commented on GitHub (May 6, 2025):

@weibingo could you please provide the result of the command: lsmod | grep nv.

@doujiang24 same problem as mine.

Here is my hardware info: (Driver Version: 535.183.06 CUDA Version: 12.6)

and output of lsmod | grep nv:

This kind of "failed to register address" error can also be reproduced by NIXL with UCX backend with test/python/nixl_api_test.py simply by changing host memory to cuda memory at line 40:

ib_md.c:282 UCX ERROR ibv_reg_mr(address=0xa02000000, length=2097152, access=0xf) failed: Bad address
ucp_mm.c:76 UCX ERROR failed to register address 0xa02000000 (cuda) length 2097152 on md[7]=mlx5_bond_0: Input/output error (md supports: host|cuda)

So I guess this might be related to RDMA driver setting or other kernel module dependency? Could you help me out? Thanks.

@bliu3650 commented on GitHub (May 6, 2025): > [@weibingo](https://github.com/weibingo) could you please provide the result of the command: `lsmod | grep nv`. @doujiang24 same problem as mine. Here is my hardware info: (Driver Version: 535.183.06 CUDA Version: 12.6) ![Image](https://github.com/user-attachments/assets/9fe9b7ce-39e3-4fbc-b028-9c97e890909e) and output of lsmod | grep nv: ![Image](https://github.com/user-attachments/assets/c39c0bb9-5360-4134-9a91-d52df46bf333) This kind of "failed to register address" error can also be reproduced by NIXL with UCX backend with [test/python/nixl_api_test.py](https://github.com/ai-dynamo/nixl/blob/0.1.0/test/python/nixl_api_test.py) simply by changing host memory to cuda memory at line 40: ![Image](https://github.com/user-attachments/assets/1b1ba2a7-e5dc-4718-9b87-82eba7419151) ib_md.c:282 UCX ERROR ibv_reg_mr(address=0xa02000000, length=2097152, access=0xf) failed: Bad address ucp_mm.c:76 UCX ERROR failed to register address 0xa02000000 (cuda) length 2097152 on md[7]=mlx5_bond_0: Input/output error (md supports: host|cuda) So I guess this might be related to RDMA driver setting or other kernel module dependency? Could you help me out? Thanks.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (May 7, 2025):

lsmod | grep nv looks good to me. @bliu3650 please contact me via wechat(id: doujiang24) if it's convenient for you.

@doujiang24 commented on GitHub (May 7, 2025): `lsmod | grep nv` looks good to me. @bliu3650 please contact me via wechat(id: doujiang24) if it's convenient for you.

bsakaguchi commented

Author

Owner

@bliu3650 commented on GitHub (May 9, 2025):

More debugging with small test case, I found an interesting difference between my system environment with yours.

Keep the default setting to run sglang v0.4.6 with mooncake, the pointer to KV gpu buffer which allocated by torch automatically starts with "0x2cbxxx" instread of "0x7fxxxx". Then the memory registeration failed with "Bad address".
Under my system environment, I create a small test to debug this "ibv_reg_mr" issue, but alloate the gpu memory using cuMemAlloc manually. As expected, the pointer to this gpu memory starts with "0x7fxxxx", and the memory registeration by ibv_reg_mr successed, except some other permission issue (ignored...).

Could anyone kindly help explain what might caused these different behaviors? Espeically why I got "0x2cbxxx" pointer from torch. Appreciated.

@bliu3650 commented on GitHub (May 9, 2025): More debugging with small test case, I found an interesting difference between my system environment with yours. 1. Keep the default setting to run sglang v0.4.6 with mooncake, the pointer to KV gpu buffer which allocated by torch automatically starts with "**0x2cbxxx**" instread of "**0x7fxxxx**". Then the memory registeration failed with "Bad address". 2. Under my system environment, I create a small test to debug this "ibv_reg_mr" issue, but alloate the gpu memory using cuMemAlloc manually. As expected, the pointer to this gpu memory starts with "**0x7fxxxx**", and the memory registeration by ibv_reg_mr successed, except some other permission issue (ignored...). Could anyone kindly help explain what might caused these different behaviors? Espeically why I got "**0x2cbxxx**" pointer from torch. Appreciated.

bsakaguchi commented

Reference:
https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html

Author

Owner

@bliu3650 commented on GitHub (May 11, 2025):

To someone who might hit similiar issue,

After replaced the default torch memory allocator with external one, all the KV buffers are now allocated with the address beginning with "0x7fxxx" which can then be successfully registered to RDMA memory region.

import torch

ext_alloc = torch.cuda.memory.CUDAPluggableAllocator(
    "<path-to-your-ext_allocator.so>",
    "ext_cu_malloc",
    "ext_cu_free"
)

torch.cuda.memory.change_current_allocator(ext_alloc)

Known issues and walkaround:

RuntimeError: Can't swap an already initialized allocator
-> walkaround: put this replacement at the begining of your applicaton
RuntimeError: CUDAPluggableAllocator does not yet support getDeviceStats
-> walkaround: remove any calling to torch.cuda.max_memory_allocated()

@bliu3650 commented on GitHub (May 11, 2025): To someone who might hit similiar issue, After replaced the default torch memory allocator with external one, all the KV buffers are now allocated with the address beginning with "0x7fxxx" which can then be successfully registered to RDMA memory region. ``` import torch ext_alloc = torch.cuda.memory.CUDAPluggableAllocator( "<path-to-your-ext_allocator.so>", "ext_cu_malloc", "ext_cu_free" ) torch.cuda.memory.change_current_allocator(ext_alloc) ``` Reference: https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html Known issues and walkaround: 1. RuntimeError: Can't swap an already initialized allocator -> walkaround: put this replacement at the begining of your applicaton 2. RuntimeError: CUDAPluggableAllocator does not yet support getDeviceStats -> walkaround: remove any calling to torch.cuda.max_memory_allocated()

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 11, 2025):

To someone who might hit similiar issue,

After replaced the default torch memory allocator with external one, all the KV buffers are now allocated with the address beginning with "0x7fxxx" which can then be successfully registered to RDMA memory region.
import torch

ext_alloc = torch.cuda.memory.CUDAPluggableAllocator(
    "<path-to-your-ext_allocator.so>",
    "ext_cu_malloc",
    "ext_cu_free"
)

torch.cuda.memory.change_current_allocator(ext_alloc)
Reference: https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html

Known issues and walkaround:

RuntimeError: Can't swap an already initialized allocator
-> walkaround: put this replacement at the begining of your applicaton

RuntimeError: CUDAPluggableAllocator does not yet support getDeviceStats
-> walkaround: remove any calling to torch.cuda.max_memory_allocated()

Thank you for your reply. I would like to know what exactly is the reason that we need to perform these additional steps?

@CSEEduanyu commented on GitHub (May 11, 2025): > To someone who might hit similiar issue, > > After replaced the default torch memory allocator with external one, all the KV buffers are now allocated with the address beginning with "0x7fxxx" which can then be successfully registered to RDMA memory region. > > ``` > import torch > > ext_alloc = torch.cuda.memory.CUDAPluggableAllocator( > "<path-to-your-ext_allocator.so>", > "ext_cu_malloc", > "ext_cu_free" > ) > > torch.cuda.memory.change_current_allocator(ext_alloc) > ``` > > Reference: https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html > > Known issues and walkaround: > > 1. RuntimeError: Can't swap an already initialized allocator > -> walkaround: put this replacement at the begining of your applicaton > 2. RuntimeError: CUDAPluggableAllocator does not yet support getDeviceStats > -> walkaround: remove any calling to torch.cuda.max_memory_allocated() Thank you for your reply. I would like to know what exactly is the reason that we need to perform these additional steps?

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 11, 2025):

What do these two different addresses represent? So is this actually a problem with the torch environment? @bliu3650

@CSEEduanyu commented on GitHub (May 11, 2025): What do these two different addresses represent? So is this actually a problem with the torch environment? @bliu3650

bsakaguchi commented

Author

Owner

@bliu3650 commented on GitHub (May 12, 2025):

What do these two different addresses represent? So is this actually a problem with the torch environment? @bliu3650

@CSEEduanyu Hi, these two addresses are both pointing to KV buffers. One is allocated by torch automatcially, another one is allocated by my external allocator. I just walkarouded to setup the entire sglang/pd flow, which was blocked by the torch mem allocator(and then memory register to RDMA) on my infra environment. There should be a problem about our hardware setting, but not the official torch, since many others in the community not report this kind of issue so far.

I'm very curious about the gpu memory address allocated by torch default behaviour (under different environment). I did not know why the address starts with 0x2cxx failed but 0x7fxx successed in this case.

@bliu3650 commented on GitHub (May 12, 2025): > What do these two different addresses represent? So is this actually a problem with the torch environment? [@bliu3650](https://github.com/bliu3650) @CSEEduanyu Hi, these two addresses are both pointing to KV buffers. One is allocated by torch automatcially, another one is allocated by my external allocator. I just walkarouded to setup the entire sglang/pd flow, which was blocked by the torch mem allocator(and then memory register to RDMA) on my infra environment. There should be a problem about our hardware setting, but not the official torch, since many others in the community not report this kind of issue so far. I'm very curious about the gpu memory address allocated by torch default behaviour (under different environment). I did not know why the address starts with 0x2cxx failed but 0x7fxx successed in this case.

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 12, 2025):

What do these two different addresses represent? So is this actually a problem with the torch environment? @bliu3650

@CSEEduanyu Hi, these two addresses are both pointing to KV buffers. One is allocated by torch automatcially, another one is allocated by my external allocator. I just walkarouded to setup the entire sglang/pd flow, which was blocked by the torch mem allocator(and then memory register to RDMA) on my infra environment. There should be a problem about our hardware setting, but not the official torch, since many others in the community not report this kind of issue so far.

I'm very curious about the gpu memory address allocated by torch default behaviour (under different environment). I did know why the address starts with 0x2xxx failed but 0x2fxx successed in this case.

Thank you for your response. Could you please provide your allocator.so library? I'd like to try and see if it can resolve the issue. @bliu3650

@CSEEduanyu commented on GitHub (May 12, 2025): > > What do these two different addresses represent? So is this actually a problem with the torch environment? [@bliu3650](https://github.com/bliu3650) > > [@CSEEduanyu](https://github.com/CSEEduanyu) Hi, these two addresses are both pointing to KV buffers. One is allocated by torch automatcially, another one is allocated by my external allocator. I just walkarouded to setup the entire sglang/pd flow, which was blocked by the torch mem allocator(and then memory register to RDMA) on my infra environment. There should be a problem about our hardware setting, but not the official torch, since many others in the community not report this kind of issue so far. > > I'm very curious about the gpu memory address allocated by torch default behaviour (under different environment). I did know why the address starts with 0x2xxx failed but 0x2fxx successed in this case. Thank you for your response. Could you please provide your allocator.so library? I'd like to try and see if it can resolve the issue. @bliu3650

bsakaguchi commented

Author

Owner

@bliu3650 commented on GitHub (May 12, 2025):

The creation of the external allocator is just the same as the one in this reference.

@bliu3650 commented on GitHub (May 12, 2025): The creation of the external allocator is just the same as the one in this [reference](https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html).

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 12, 2025):

The creation of the external allocator is just the same as the one in this reference.

Great, this really works

@CSEEduanyu commented on GitHub (May 12, 2025): > The creation of the external allocator is just the same as the one in this [reference](https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html). Great, this really works

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 15, 2025):

The creation of the external allocator is just the same as the one in this reference.

This method indeed no longer throws an error, but I found that it causes a significant performance drop @bliu3650

@CSEEduanyu commented on GitHub (May 15, 2025): > The creation of the external allocator is just the same as the one in this [reference](https://kshitij12345.github.io/python,/pytorch/2023/02/26/External-CUDA-Allocator-With-PyTorch.html). This method indeed no longer throws an error, but I found that it causes a significant performance drop @bliu3650

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 15, 2025):

@RockGo @orrorcol Have you completely resolved this issue?

@CSEEduanyu commented on GitHub (May 15, 2025): @RockGo @orrorcol Have you completely resolved this issue?

bsakaguchi commented

Author

Owner

@CSEEduanyu commented on GitHub (May 16, 2025):

I found that the root cause of this problem is that the kv_cache address allocated by native torch.zero() is different from the kv_cache address allocated by cuda_malloc(). The former will cause an error of bad address when calling ibv_reg_mr(), while the latter will not, but it will cause sglang's throughput to decrease. @doujiang24 @ShangmingCai I contacted SRE, but it seems they don't have much information either. Could you please advise me on how to troubleshoot this issue?

@CSEEduanyu commented on GitHub (May 16, 2025): I found that the root cause of this problem is that the kv_cache address allocated by native torch.zero() is different from the kv_cache address allocated by cuda_malloc(). The former will cause an error of bad address when calling ibv_reg_mr(), while the latter will not, but it will cause sglang's throughput to decrease. @doujiang24 @ShangmingCai I contacted SRE, but it seems they don't have much information either. Could you please advise me on how to troubleshoot this issue?

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (May 16, 2025):

I found that the root cause of this problem is that the kv_cache address allocated by native torch.zero() is different from the kv_cache address allocated by cuda_malloc(). The former will cause an error of bad address when calling ibv_reg_mr(), while the latter will not, but it will cause sglang's throughput to decrease. @doujiang24 @ShangmingCai I contacted SRE, but it seems they don't have much information either. Could you please advise me on how to troubleshoot this issue?

@CSEEduanyu We don't allocate GPU mem buffer in the mooncake layer, we merely register it. The allocation is done by sglang, maybe you could report this to the sglang repo to see if there are some parameters or env vars that you could use for this scenario.

@ShangmingCai commented on GitHub (May 16, 2025): > I found that the root cause of this problem is that the kv_cache address allocated by native torch.zero() is different from the kv_cache address allocated by cuda_malloc(). The former will cause an error of bad address when calling ibv_reg_mr(), while the latter will not, but it will cause sglang's throughput to decrease. [@doujiang24](https://github.com/doujiang24) [@ShangmingCai](https://github.com/ShangmingCai) I contacted SRE, but it seems they don't have much information either. Could you please advise me on how to troubleshoot this issue? @CSEEduanyu We don't allocate GPU mem buffer in the mooncake layer, we merely register it. The allocation is done by sglang, maybe you could report this to the sglang repo to see if there are some parameters or env vars that you could use for this scenario.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (May 16, 2025):

@CSEEduanyu It maybe also be related to the machines' setup (please check this with your SRE), one of the machines does not properly set up the GDR env, so the torch will use another impl to allocate the buffer.

@ShangmingCai commented on GitHub (May 16, 2025): @CSEEduanyu It maybe also be related to the machines' setup (please check this with your SRE), one of the machines does not properly set up the GDR env, so the torch will use another impl to allocate the buffer.

bsakaguchi commented

Author

Owner

@github-actions[bot] commented on GitHub (Jul 14, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 14, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented