turboderp-org/exllamav3
1
0
Fork 0
mirror of https://github.com/turboderp-org/exllamav3.git synced 2026-04-20 14:29:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

ppl_transformers.py: Explicitly make bfloat16 the default dtype

Browse Source
This commit is contained in:
turboderp
2025-09-18 22:11:19 +02:00
parent 476e591966
commit 3845775650
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
eval/ppl_transformers.py
View File

@@ -48,7 +48,7 @@ def main(args):
model = AutoModelForCausalLM.from_pretrained(
args.model_dir,
device_map = "auto",
torch_dtype = torch.half if args.tight else torch.float if args.fp32 else None,
torch_dtype = torch.half if args.tight else torch.float if args.fp32 else torch.bfloat16,
)
if args.tight:
free_mem()