mirror of
https://github.com/kvcache-ai/sglang.git
synced 2026-07-08 16:27:09 +00:00
50 lines
2.4 KiB
Markdown
50 lines
2.4 KiB
Markdown
# R-Fork
|
|
|
|
R-Fork (Tensor Remote Fork) is a novel weight loading methodology that leverages efficient inter-node GPU-to-GPU data transfer path to load tensors from a running SGLang instance to a new instance with zero-copy. It can significantly optimize the SGLang instance boot-up time by reducing model weights loading from several minutes to mere seconds.
|
|
|
|
To learn more details about R-Fork, please check **<a href=https://lmsys.org/blog/2025-12-10-rfork/> R-Fork blog </a>**
|
|
|
|
## Usage
|
|
|
|
| Argument | Usage |
|
|
|--------------|--------------------------------------------|
|
|
| load-format | set to `remote_instance` to enable R-Fork. |
|
|
| remote-instance-weight-loader-backend | `nccl` or `transfer_engine`, default value is `nccl` |
|
|
| remote-instance-weight-loader-seed-instance-ip | IP address of the seed instance who will provide the model weight |
|
|
| remote-instance-weight-loader-seed-instance-service-port | the port that the seed instance's HTTP server is listening on |
|
|
| remote-instance-weight-loader-send-weights-group-ports | the list of available ports on the seed instance that will be used to build NCCL communication groups between seed and client instance. This argument is only needed by `nccl` backend. |
|
|
| remote-instance-weight-loader-start-seed-via-transfer-engine | set to start seed service that supports TransferEngine as backend. It is needed for seed instances when using `transfer_engine` as backend. |
|
|
|
|
### NCCL as backend
|
|
|
|
seed instance:
|
|
```shell
|
|
python -m sglang.launch_server [args]
|
|
```
|
|
|
|
client instance:
|
|
```shell
|
|
python -m sglang.launch_server [args] \
|
|
--load-format remote_instance \
|
|
--remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
|
|
--remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
|
|
--remote-instance-weight-loader-send-weights-group-ports [send_weights_nccl_group_ports_list] \
|
|
--remote-instance-weight-loader-backend nccl
|
|
```
|
|
|
|
### TransferEngine as backend
|
|
|
|
seed instance:
|
|
```shell
|
|
python -m sglang.launch_server [args] \
|
|
--remote-instance-weight-loader-start-seed-via-transfer-engine
|
|
```
|
|
|
|
```shell
|
|
python -m sglang.launch_server [args] \
|
|
--load-format remote_instance \
|
|
--remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
|
|
--remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
|
|
--remote-instance-weight-loader-backend transfer_engine
|
|
```
|