ai-toolkit

ostris/ai-toolkit

Fork 0

mirror of https://github.com/ostris/ai-toolkit.git synced 2026-02-02 03:49:47 +00:00

Commit Graph

Select branches

Hide Pull Requests

WIP

dev

development

kohya-sdxl

lumina2

main

multi-gpu

qwen_image_edit

sdxl

wavelet_loss

#107

#109

#115

#12

#120

#124

#125

#128

#128

#138

#138

#141

#143

#155

#155

#156

#156

#158

#158

#160

#160

#168

#168

#173

#176

#177

#179

#179

#183

#184

#184

#185

#185

#191

#191

#196

#224

#224

#226

#226

#232

#233

#233

#244

#244

#245

#245

#246

#246

#249

#25

#25

#251

#251

#253

#253

#256

#263

#263

#264

#265

#265

#271

#273

#273

#274

#275

#275

#278

#283

#283

#290

#290

#300

#300

#302

#302

#303

#303

#304

#307

#318

#318

#326

#326

#331

#331

#333

#333

#341

#341

#342

#342

#343

#350

#350

#353

#353

#354

#354

#359

#362

#362

#373

#373

#375

#375

#377

#381

#381

#382

#382

#383

#385

#385

#392

#392

#395

#407

#407

#41

#417

#418

#420

#420

#422

#426

#427

#427

#430

#430

#431

#431

#432

#432

#434

#442

#442

#446

#447

#452

#455

#459

#459

#461

#461

#464

#465

#465

#467

#469

#469

#473

#473

#474

#488

#488

#489

#489

#492

#492

#494

#494

#496

#498

#498

#499

#499

#500

#509

#511

#513

#519

#519

#520

#520

#521

#522

#523

#525

#527

#527

#540

#547

#547

#551

#551

#555

#558

#559

#559

#563

#563

#564

#564

#567

#578

#578

#580

#580

#581

#581

#583

#594

#599

#60

#600

#600

#601

#601

#606

#606

#608

#608

#61

#610

#611

#614

#614

#616

#616

#629

#629

#630

#630

#633

#633

#637

#637

#639

#639

#644

#646

#648

#649

#649

#655

#655

#658

#658

#660

#661

#661

#666

#666

#668

#668

#673

#673

#682

#685

#685

#687

#687

#69

#69

#692

#692

#694

#694

#695

#695

#7

#80

#86

#86

#9

#95

4aa19b5c1d Only quantize flux T5 is also quantizing model. Load TE from original name and path if fine tuning. Jaret Burkett 2024-10-29 14:25:31 -06:00
4747716867 Fixed issue with adapters not providing gradients with new grad activator Jaret Burkett 2024-10-29 14:22:10 -06:00
22cd40d7b9 Improvements for full tuning flux. Added debugging launch config for vscode Jaret Burkett 2024-10-29 04:54:08 -06:00
3400882a80 Added preliminary support for SD3.5-large lora training Jaret Burkett 2024-10-22 12:21:36 -06:00
9f94c7b61e Added experimental param multiplier to the ema module Jaret Burkett 2024-10-22 09:25:52 -06:00
bedb8197a2 Fixed issue with sizes for some images being loaded sideways resulting in squished images. Jaret Burkett 2024-10-20 11:51:29 -06:00
e3ebd73610 Add a projection layer on vision direct when doing image embeds Jaret Burkett 2024-10-20 10:48:23 -06:00
dd931757cd Merge branch 'main' of github.com:ostris/ai-toolkit Jaret Burkett 2024-10-20 07:04:29 -06:00
0640cdf569 Handle errors in loading size database Jaret Burkett 2024-10-20 07:04:19 -06:00
0b048d0dde Locked version of quanto as it breaks in later versions Jaret Burkett 2024-10-16 22:41:04 +00:00
473d455f44 Process empty clip image if there is not one for reg images when training a custom adapter Jaret Burkett 2024-10-15 08:28:04 -06:00
ce759ebd8c Normalize the image embeddings on vd adapter forward Jaret Burkett 2024-10-12 15:09:48 +00:00
628a7923a3 Remove norm on image embeds on custom adapter Jaret Burkett 2024-10-12 00:43:18 +00:00
3922981996 Added some additional experimental things to the vision direct encoder Jaret Burkett 2024-10-10 19:42:26 +00:00
ab22674980 Allow for a default caption file in the folder. Minor bug fixes. Jaret Burkett 2024-10-10 07:31:33 -06:00
9452929300 Apply a mask to the embeds for SD if using T5 encoder Jaret Burkett 2024-10-04 10:55:20 -06:00
a800c9d19e Add a method to have an inference only lora Jaret Burkett 2024-10-04 10:06:53 -06:00
28e6f00790 Fixed bug in returning clip image embed to actually return it Jaret Burkett 2024-10-03 10:49:09 -06:00
67e0aca750 Added ability to load clip pairs randomly from folder. Other small bug fixes Jaret Burkett 2024-10-03 10:03:49 -06:00
f05224970f Added Vision Languate Adapter usage for pixtral vd adapter Jaret Burkett 2024-09-29 19:39:56 -06:00
b4f64de4c2 Quick patch to scope xformer imports until a better solution Jaret Burkett 2024-09-28 15:36:42 -06:00
2e5f6668dc Add xformers ad a dependency Jaret Burkett 2024-09-28 15:30:14 -06:00
e4c82803e1 Handle random resizing for pixtral input on direct vision adapter Jaret Burkett 2024-09-28 14:53:38 -06:00
69aa92bce5 Added support for AdEMAMix8bit Jaret Burkett 2024-09-28 14:33:51 -06:00
a508caad1d Change pixtral to crop based on number of pixels instead of largest dimension Jaret Burkett 2024-09-28 13:05:26 -06:00
58537fc92b Added initial direct vision pixtral support Jaret Burkett 2024-09-28 10:47:51 -06:00
86b5938cf3 Fixed the webp bug finally. Jaret Burkett 2024-09-25 13:56:00 -06:00
6b4034122f REmove layers from direct vision resampler Jaret Burkett 2024-09-24 15:08:29 -06:00
10817696fb Fixed issue where direct vision was not passing additional params from resampler when it is added Jaret Burkett 2024-09-24 10:34:11 -06:00
037ce11740 Always return vision encoder in state dict Jaret Burkett 2024-09-24 07:43:17 -06:00
04424fe2d6 Added config setting to set the timestep type Jaret Burkett 2024-09-24 06:53:59 -06:00
40a8ff5731 Load local hugging face packages for assistant adapter Jaret Burkett 2024-09-23 10:37:12 -06:00
2776221497 Added option to cache empty prompt or trigger and unload text encoders while training Jaret Burkett 2024-09-21 20:54:09 -06:00
f85ad452c6 Added initial support for pixtral vision as a vision encoder. Jaret Burkett 2024-09-21 15:21:14 -06:00
dd889086f4 Updates to the docker file for jupyterlab Jaret Burkett 2024-09-21 12:07:07 -06:00
bc693488eb fix diffusers codebase (#183) apolinário 2024-09-21 12:50:29 -05:00
d97c55cd96 Updated requirements to lock version of albucore, which had breaking changes. Jaret Burkett 2024-09-21 11:19:13 -06:00
79b4e04b80 Feat: Wandb logging (#95) Plat 2024-09-20 11:01:01 +09:00
951e223481 Added support to disable single transformers in vision direct adapter Jaret Burkett 2024-09-11 08:54:51 -06:00
fc34a69bec Ignore guidance embed when full tuning flux. adjust block scaler to decat to 1.0. Add MLP resampler for reducing vision adapter tokens Jaret Burkett 2024-09-09 16:24:46 -06:00
279ee65177 Remove block scaler Jaret Burkett 2024-09-06 08:28:17 -06:00
3a1f464132 Added support for training vision direct weight adapters Jaret Burkett 2024-09-05 10:11:44 -06:00
5c8fcc8a4e Fix bug with zeroing out gradients when accumulating Jaret Burkett 2024-09-03 08:29:15 -06:00
121a760c19 Added proper grad accumulation Jaret Burkett 2024-09-03 07:24:18 -06:00
e5fadddd45 Added ability to do prompt attn masking for flux Jaret Burkett 2024-09-02 17:29:36 -06:00
d44d4eb61a Added a new experimental linear weighing technique Jaret Burkett 2024-09-02 09:22:13 -06:00
7d9ab22405 Rework ip adapter and vision direct adapters to apply to the single transformer blocks even though they are not cross attn. Jaret Burkett 2024-09-01 10:40:42 -06:00
7ed8c51f20 Readme cleanup Jaret Burkett 2024-09-01 07:06:09 -06:00
6df33156f0 Add information about specific weight targeting in the README Jaret Burkett 2024-09-01 06:59:47 -06:00
40f5c59da0 Fixes for training ilora on flux Jaret Burkett 2024-08-31 16:55:26 -06:00
3e71a99df0 Check for contains only against clean name for lora, not the adjusted one Jaret Burkett 2024-08-31 07:44:13 -06:00
562405923f Update README.md for push_to_hub (#143) apolinário 2024-08-30 17:34:28 -05:00
f84bd6d7a6 Add Gradio UI for ai-toolkit (#141) apolinário 2024-08-30 07:29:51 -05:00
4fa8fac5fd WIP multidevice training multi-gpu Jaret Burkett 2024-08-29 16:04:20 -06:00
a48c9aba8d Created a v2 trainer and moved all the training logic to single torch model so it can can be run in parallel Jaret Burkett 2024-08-29 12:34:18 -06:00
60232def91 Made peleminary arch for flux ip adapter training Jaret Burkett 2024-08-28 08:55:39 -06:00
3843e0d148 Added support for vision direct adapter for flux Jaret Burkett 2024-08-26 16:27:28 -06:00
e127c079da fix: print out the path where the image encode failed (#107) liaoliaojun 2024-08-23 11:34:35 +08:00
34db804c76 Modal cloud training support, fixed typo in toolkit/scheduler.py, Schnell training support for Colab, issue #92 , issue #114 (#115) martintomov 2024-08-23 06:25:44 +03:00
4d35a29c97 Add push_to_hub to the trainer (#109) apolinário 2024-08-23 04:18:56 +01:00
b322d05fa3 Added tutorial link to readme Jaret Burkett 2024-08-22 16:25:32 -06:00
8577849eeb Fixed wrong discord link. Woops. Jaret Burkett 2024-08-22 14:49:03 -06:00
338c77d677 Fixed breaking change with diffusers. Allow flowmatch on normal stable diffusion models. Jaret Burkett 2024-08-22 14:36:22 -06:00
e07a98a50c Bugfixes for full finetuning at bf16 Jaret Burkett 2024-08-22 05:15:33 -06:00
6a754b2710 Merge branch 'main' of github.com:ostris/ai-toolkit Jaret Burkett 2024-08-22 04:36:50 -06:00
a939cf3730 WIP - adding support for flux DoRA and ip adapter training Jaret Burkett 2024-08-22 04:36:39 -06:00
169dbd22ba Finaized bug reports Jaret Burkett 2024-08-18 16:21:48 -06:00
6e7d721382 More issues testing Jaret Burkett 2024-08-18 16:20:08 -06:00
dc6f36cd82 Testing github bug reporting stuff Jaret Burkett 2024-08-18 16:09:52 -06:00
5603f9e004 issue #76, and RunPod cloud training setup #90 (#80) martintomov 2024-08-19 00:43:45 +03:00
c45887192a Unload interum weights when doing multi lora fuse Jaret Burkett 2024-08-18 09:35:10 -06:00
13a965a26c Fixed bad key naming on lora fuse I just pushed Jaret Burkett 2024-08-18 09:33:31 -06:00
77ee7090e8 Update FAQ.md Jaret Burkett 2024-08-18 09:26:22 -06:00
078396ceac Added a basic FAQ Jaret Burkett 2024-08-18 09:21:51 -06:00
f944eeaa4d Fuse flux schnell assistant adapter in pieces when doing lowvram to drastically speed ip up from minutes to seconds. Jaret Burkett 2024-08-18 09:09:11 -06:00
81899310f8 Added support for training on flux schnell. Added example config and instructions for training on flux schnell Jaret Burkett 2024-08-17 06:58:39 -06:00
f9179540d2 Flush after sampling Jaret Burkett 2024-08-16 17:29:42 -06:00
452e0e286d For lora assisted training, merge in before quantizing then sample with schnell at -1 weight. Almost doubles training speed with lora adapter. Jaret Burkett 2024-08-16 17:28:44 -06:00
165510ace2 Dumb typo Jaret Burkett 2024-08-15 12:59:32 -06:00
0355662e8e Added support for polarity guidance for flow matching models Jaret Burkett 2024-08-15 12:22:00 -06:00
b99d36dfdb fixed issue with batch sizes larget than 1 Jaret Burkett 2024-08-15 12:21:38 -06:00
9001e5c933 Change flux latent spact if so it will not use old cache Jaret Burkett 2024-08-14 11:27:40 -06:00
7fed4ea761 fixed huge flux training bug. Added ability to use an assistatn lora Jaret Burkett 2024-08-14 10:14:13 -06:00
e07bf11727 Merge pull request #61 from fofr/patch-1 Jaret Burkett 2024-08-14 08:01:51 -06:00
c728cc9a0b Update README.md fofr 2024-08-14 15:00:02 +01:00
00bd3d54a3 Actually use the save dtype from the config file. Jaret Burkett 2024-08-13 17:08:27 -06:00
f7cf2f866f Make 100% sure lora alpha matches for flux Jaret Burkett 2024-08-13 14:24:03 -06:00
465bc1e2f8 Update readme again Jaret Burkett 2024-08-13 13:37:22 -06:00
0beca0d4a7 Updated readme Jaret Burkett 2024-08-13 13:35:20 -06:00
418f5f7e8c Added new experimental time step weighing that should solve a lot of issues with distribution. Updated example. Removed a warning Jaret Burkett 2024-08-13 12:02:11 -06:00
9ee1ef2a0a Added experimental modified sigma sqrt weight mapping for linear timestep scheduling for flowmatching Jaret Burkett 2024-08-12 17:03:09 -06:00
599fafe01f Allow user to have the full flux checkpoint local Jaret Burkett 2024-08-12 09:57:16 -06:00
af108bb964 Bug fix with dataloader. Added a flag to completly disable sampling Jaret Burkett 2024-08-12 09:19:40 -06:00
89d61a3b8e Readme updates Jaret Burkett 2024-08-11 13:23:57 -06:00
a6aa4b2c7d Added ability to set timesteps to linear for flowmatching schedule Jaret Burkett 2024-08-11 13:06:08 -06:00
f8f0657b68 Added a colab notebook for training flux loras Jaret Burkett 2024-08-11 12:27:40 -06:00
7f0ecdb377 Merge branch 'main' of github.com:ostris/ai-toolkit Jaret Burkett 2024-08-11 11:10:45 -06:00
fbed8568fb Actually use the correct timestep sampling instead of calculating it and moving on lol. Tested a few with it and it seems to work better. Jaret Burkett 2024-08-11 11:10:37 -06:00
6d31c6db73 Added a fix for windows dataloader Jaret Burkett 2024-08-11 10:48:24 -06:00
6490a326e5 Fixed issue for vaes without a shift Jaret Burkett 2024-08-11 10:30:55 -06:00