mirror of
https://github.com/kvcache-ai/sglang.git
synced 2026-07-13 10:56:58 +00:00
a3291b5654
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
103 lines
2.7 KiB
Plaintext
103 lines
2.7 KiB
Plaintext
---
|
|
title: NVIDIA Jetson Orin
|
|
description: Guide for installing and running SGLang on NVIDIA Jetson Orin devices.
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
Before starting, ensure the following:
|
|
|
|
- [NVIDIA Jetson AGX Orin Devkit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with JetPack 6.1 or later.
|
|
- CUDA Toolkit and cuDNN are installed.
|
|
- Verify that the Jetson AGX Orin is in high-performance mode:
|
|
|
|
<CodeGroup>
|
|
```bash
|
|
sudo nvpmodel -m 0
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Installing and Running SGLang with Jetson Containers
|
|
|
|
1. **Clone the jetson-containers repository**
|
|
|
|
```bash
|
|
git clone https://github.com/dusty-nv/jetson-containers.git
|
|
```
|
|
|
|
2. **Run the installation script**
|
|
|
|
```bash
|
|
bash jetson-containers/install.sh
|
|
```
|
|
|
|
3. **Build the container image**
|
|
|
|
```bash
|
|
jetson-containers build sglang
|
|
```
|
|
|
|
4. **Run the container**
|
|
|
|
<Tabs>
|
|
<Tab title="Using jetson-containers">
|
|
```bash
|
|
jetson-containers run $(autotag sglang)
|
|
```
|
|
</Tab>
|
|
<Tab title="Using Docker manually">
|
|
```bash
|
|
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
## Running Inference
|
|
|
|
Launch the server:
|
|
|
|
<CodeGroup>
|
|
```bash
|
|
python -m sglang.launch_server \
|
|
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
|
|
--device cuda \
|
|
--dtype half \
|
|
--attention-backend flashinfer \
|
|
--mem-fraction-static 0.8 \
|
|
--context-length 8192
|
|
```
|
|
</CodeGroup>
|
|
|
|
The quantization and limited context length (`--dtype half` `--context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).
|
|
|
|
After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#Usage) to test the usability.
|
|
|
|
## Running Quantization with TorchAO
|
|
|
|
TorchAO is suggested to NVIDIA Jetson Orin.
|
|
|
|
<CodeGroup>
|
|
```bash
|
|
python -m sglang.launch_server \
|
|
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
|
--device cuda \
|
|
--dtype bfloat16 \
|
|
--attention-backend flashinfer \
|
|
--mem-fraction-static 0.8 \
|
|
--context-length 8192 \
|
|
--torchao-config int4wo-128
|
|
```
|
|
</CodeGroup>
|
|
|
|
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
|
|
|
|
## Structured Output with XGrammar
|
|
|
|
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).
|
|
|
|
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
|
|
|
|
## References
|
|
|
|
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
|