turboderp-org/exllamav2
1
0
Fork 0
mirror of https://github.com/turboderp-org/exllamav2.git synced 2026-04-20 14:29:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
turboderp
2023-09-12 06:52:04 +02:00
parent 546c91482e
commit 8e3cd01889
1 changed files with 6 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -56,11 +56,15 @@ models and various other finetunes. You can also provide a custom system prompt
### Installation
Clone the repository and run `python setup.py install --user`. (PyPi package is coming, be patient.)
To install as a library (not required for the included examples), clone the repository and run:
```
python setup.py install --user`.
```
ExLlamaV2 relies on a Torch C++ extension for its CUDA functions, which is compiled at runtime. This means the first
time the library is used it will take 10-20 seconds (depending on your hardware) to start, but the extension gets cached
for subsequent use.
for subsequent use. A PyPI package will evantually be available with an option to install a precompiled extension.
## EXL2 quantization