rattus d297a749a2 dynamic_vram: Fix windows Aimdo crash + Fix LLM performance (#12408) ...

* model_management: lazy-cache aimdo_tensor

These tensors cosntructed from aimdo-allocations are CPU expensive to
make on the pytorch side. Add a cache version that will be valid with
signature match to fast path past whatever torch is doing.

* dynamic_vram: Minimize fast path CPU work

Move as much as possible inside the not resident if block and cache
the formed weight and bias rather than the flat intermediates. In
extreme layer weight rates this adds up.

2026-02-11 14:50:16 -05:00

41 KiB

Raw Blame History

View Raw