[GH-ISSUE #70] Win32 FILE_FLAG_SEQUENTIAL_SCAN has a feature/side effect of discarding data from OS cache after first access #22

Open
opened 2026-10-26 22:23:42 +00:00 by bsakaguchi · 24 comments
Owner

Originally created by @brendanhoar on GitHub (Jun 29, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/70

Presumably this is for @rattus128's attention, correct me if I am wrong.

On windows 32, aimdo's management of safetensors files opens the handle with FILE_FLAG_SEQUENTIAL_SCAN, presumably for performance reasons.

However, a side effect, or feature, depending upon your perspective is that, with this flag, the OS CACHE only caches the data pulled from the file until it is read once, and then it discards the cached data immediately.

In comfyui, I had hoped that --high-ram would have an impact here, but it doesn't seem to change the file handle creation logistics, and that's why I continue to see massive data reads if the models iterated through a single workflow generation use more VRAM than I have (24GB), even though I have 512GB RAM.

Would it be possible to expose controls on whether this flag is used or not, potentially also, whether FILE_FLAG_OVERLAPPED should also be controlled, as it too may have an impact on windows internal OS caching strategies (I haven't looked deep into this yet, but also, I am concerned about out of order IO making it down to physical devices...which may or may not be related)

Presumably these two flags, among others, were used for fastest initial load, perhaps trying to bypass some of the horrible performance issues seen in the past with out of order faulting causing disk thrash, but ... certainly there's a way to ensure that that is dealt with while also not throwing away OS-cached data on systems built specifically to take advantage of them?

Originally, I was investigating FILE_SHARED_WRITE (unsure why this is used) but ran into the above, which appears to be the real root cause of performance issues on my system,

In the end what I would think would be preferable is both allowing the USER with lots of RAM to cache as much safetensors file data as possible at the OS level, while also ensuring that data access patterns to the mapped memory are both large in size and in linear order, so that faults to slower storage don't suddenly incur massive time and hardware thrashing penalties.

Thanks.
B

Originally created by @brendanhoar on GitHub (Jun 29, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/70 Presumably this is for @rattus128's attention, correct me if I am wrong. On windows 32, aimdo's management of safetensors files opens the handle with FILE_FLAG_SEQUENTIAL_SCAN, presumably for performance reasons. However, a side effect, or feature, depending upon your perspective is that, with this flag, the OS CACHE only caches the data pulled from the file until it is read once, and then it discards the cached data immediately. In comfyui, I had hoped that --high-ram would have an impact here, but it doesn't seem to change the file handle creation logistics, and that's why I continue to see massive data reads if the models iterated through a single workflow generation use more VRAM than I have (24GB), even though I have 512GB RAM. Would it be possible to expose controls on whether this flag is used or not, potentially also, whether FILE_FLAG_OVERLAPPED should also be controlled, as it too may have an impact on windows internal OS caching strategies (I haven't looked deep into this yet, but also, I am concerned about out of order IO making it down to physical devices...which may or may not be related) Presumably these two flags, among others, were used for fastest initial load, perhaps trying to bypass some of the horrible performance issues seen in the past with out of order faulting causing disk thrash, but ... certainly there's a way to ensure that that is dealt with while also not throwing away OS-cached data on systems built specifically to take advantage of them? Originally, I was investigating FILE_SHARED_WRITE (unsure why this is used) but ran into the above, which appears to be the real root cause of performance issues on my system, In the end what I would think would be preferable is both allowing the USER with lots of RAM to cache as much safetensors file data as possible at the OS level, while also ensuring that data access patterns to the mapped memory are both large in size and in linear order, so that faults to slower storage don't suddenly incur massive time and hardware thrashing penalties. Thanks. B
Author
Owner

@VladiCz commented on GitHub (Jun 30, 2026):

I'm on Ubuntu 24.04 and I experience slowdown since ComfyUI 0.22.0 if I remember correctly. My RAM is mostly free, but unloading LTX-2.3 model from VRAM to make space for text-encoder, video decode VAE, audio decode VAE (which is extremely VRAM hungry compared to video VAE), the unloaded model then loads in VRAM very slow, like 100-200 MB/s. This is with 128 GB DDR5 mostly empty and nvme SSD exclusively used for model data files only. I use Lora in every workflow, so maybe Lora have effect on how/from where the model is loaded, not sure.

I added the new comfyui startup option --high-ram, it really eats much more memory, but the speed is identical as without that option, it still loads unloaded model blocks 100-200 MB/s. This is my reading from KJ MemoryVisualisation.

I even disabled barrier (fstab mount option barrier=0) for that nvme model drive to make sure it is not affected by a known Ubuntu SSDs slowdown bug.

<!-- gh-comment-id:4843009403 --> @VladiCz commented on GitHub (Jun 30, 2026): I'm on Ubuntu 24.04 and I experience slowdown since ComfyUI 0.22.0 if I remember correctly. My RAM is mostly free, but unloading LTX-2.3 model from VRAM to make space for text-encoder, video decode VAE, audio decode VAE (which is extremely VRAM hungry compared to video VAE), the unloaded model then loads in VRAM very slow, like 100-200 MB/s. This is with 128 GB DDR5 mostly empty and nvme SSD exclusively used for model data files only. I use Lora in every workflow, so maybe Lora have effect on how/from where the model is loaded, not sure. I added the new comfyui startup option --high-ram, it really eats much more memory, but the speed is identical as without that option, it still loads unloaded model blocks 100-200 MB/s. This is my reading from KJ MemoryVisualisation. I even disabled barrier (fstab mount option barrier=0) for that nvme model drive to make sure it is not affected by a known Ubuntu SSDs slowdown bug.
Author
Owner

@brendanhoar commented on GitHub (Jun 30, 2026):

@VladiCz -

Be careful with MemoryVisualization, when installed the default options automatically purge models after each run of a workflow (at least, that's what it was set to on my system).

I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the loading semantics. I was trying to use it a few weeks ago to ascertain the RE-loading semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow.

I take all the above back w/r/t my comments on MemoryVisualization tool, what I saw was from the KJ Node VRAM Debugger. Apologies to @VladiCz.

In any case, let's keep focus on aimdo specifically telling Windows OS to discard the safetensors file data from the OS cache as a side effect of the FILE_FLAG_SEQUENTIAL_SCAN flag on CreateFileW.

<!-- gh-comment-id:4843698051 --> @brendanhoar commented on GitHub (Jun 30, 2026): @VladiCz - > > Be careful with MemoryVisualization, when installed the default options *automatically purge models* after each run of a workflow (at least, that's what it was set to on my system). > > I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the _loading_ semantics. I was trying to use it a few weeks ago to ascertain the _RE-loading_ semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow. I take all the above back w/r/t my comments on MemoryVisualization tool, what I saw was from the KJ Node VRAM Debugger. Apologies to @VladiCz. In any case, let's keep focus on aimdo specifically telling Windows OS to discard the safetensors file data from the OS cache as a side effect of the FILE_FLAG_SEQUENTIAL_SCAN flag on CreateFileW.
Author
Owner

@VladiCz commented on GitHub (Jun 30, 2026):

@VladiCz - Be careful with MemoryVisualization, when installed the default options automatically purge models after each run of a workflow (at least, that's what it was set to on my system).

I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the loading semantics. I was trying to use it a few weeks ago to ascertain the RE-loading semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow.

That would explain all the issues I have with unloading. But @rattus128 usually requests the output from that node for reporting issues with memory, but that would produce wrong results. If what you write is true, that would be a big issue. I'm going to see the if I will find the default setting to unload models.

<!-- gh-comment-id:4845787913 --> @VladiCz commented on GitHub (Jun 30, 2026): > [@VladiCz](https://github.com/VladiCz) - Be careful with MemoryVisualization, when installed the default options _automatically purge models_ after each run of a workflow (at least, that's what it was set to on my system). > > I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the _loading_ semantics. I was trying to use it a few weeks ago to ascertain the _RE-loading_ semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow. That would explain all the issues I have with unloading. But @rattus128 usually requests the output from that node for reporting issues with memory, but that would produce wrong results. If what you write is true, that would be a big issue. I'm going to see the if I will find the default setting to unload models.
Author
Owner

@VladiCz commented on GitHub (Jun 30, 2026):

@VladiCz - Be careful with MemoryVisualization, when installed the default options automatically purge models after each run of a workflow (at least, that's what it was set to on my system).

I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the loading semantics. I was trying to use it a few weeks ago to ascertain the RE-loading semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow.

@brendanhoar
So, what you wrote did not make much sense to me. MemoryVisualisation really should not do what you wrote. There is not "default" option to unload models. There is only a single button to unload manually, but that is not what you ment. Could you please send a screenshot of the setting you found as I cannot find any setting for automatic model unloading.

Image

Button to manually unload (not automatic):
Image

<!-- gh-comment-id:4845848871 --> @VladiCz commented on GitHub (Jun 30, 2026): > [@VladiCz](https://github.com/VladiCz) - Be careful with MemoryVisualization, when installed the default options _automatically purge models_ after each run of a workflow (at least, that's what it was set to on my system). > > I don't know why the default settings of that node pack are done that way...it's ironic that a tool people are using to try to figure out what is going on...actively changes the mechanics of what happens, usually in a pessimal way. Likely KJ is using it to look at the _loading_ semantics. I was trying to use it a few weeks ago to ascertain the _RE-loading_ semantics, but it actively interferes unless you changes the default options every time you open or regenerate a workflow. @brendanhoar So, what you wrote did not make much sense to me. MemoryVisualisation really should not do what you wrote. There is not "default" option to unload models. There is only a single button to unload manually, but that is not what you ment. Could you please send a screenshot of the setting you found as I cannot find any setting for automatic model unloading. <img width="297" height="243" alt="Image" src="https://github.com/user-attachments/assets/e6b1c2e8-1c0d-42c8-9cd7-0639e3fcb3d4" /> Button to manually unload (not automatic): <img width="189" height="95" alt="Image" src="https://github.com/user-attachments/assets/c73971fa-2769-4a83-b9ec-b3a3f09287dd" />
Author
Owner

@brendanhoar commented on GitHub (Jun 30, 2026):

There's a node added to the workflow under some conditions, let me go it. Ah, it's this one.

Sometimes this was being added to workflows inline without me doing it manually (consciously). I'm not sure why (startup flags? on detect of multimodel workflows? me clicking somewhere on the visualizer?), but I glanced over once and saw all three of these options were enabled, which...obviously made things slower. So I uninstalled the visualizer.

Image

Edit: looks like that node is just part of KJNodes. No idea why it kept getting inserted into workflows. So, nevermind...sorry for sending you down the wrong path? Anyway, back on topic to the CreateFileW flags issue.

<!-- gh-comment-id:4847192978 --> @brendanhoar commented on GitHub (Jun 30, 2026): There's a node added to the workflow under some conditions, let me go it. Ah, it's this one. Sometimes this was being added to workflows inline without me doing it manually (consciously). I'm not sure why (startup flags? on detect of multimodel workflows? me clicking somewhere on the visualizer?), but I glanced over once and saw all three of these options were enabled, which...obviously made things slower. So I uninstalled the visualizer. <img width="238" height="210" alt="Image" src="https://github.com/user-attachments/assets/4fff5774-f758-4869-afa8-e739310dc948" /> Edit: looks like that node is just part of KJNodes. No idea why it kept getting inserted into workflows. So, nevermind...sorry for sending you down the wrong path? Anyway, back on topic to the CreateFileW flags issue.
Author
Owner

@VladiCz commented on GitHub (Jul 1, 2026):

@brendanhoar The MemoryVisualiser is very handy tool so I recommend using all the time as you already discovered, that VRAM Debug from KJ nodes is a very different thing many user will never encounter or only by mistake.

<!-- gh-comment-id:4849839891 --> @VladiCz commented on GitHub (Jul 1, 2026): @brendanhoar The MemoryVisualiser is very handy tool so I recommend using all the time as you already discovered, that VRAM Debug from KJ nodes is a very different thing many user will never encounter or only by mistake.
Author
Owner

@Aamir3d commented on GitHub (Jul 11, 2026):

cc @comfyanonymous and @Kosinkadink - FYI and not meaning to distract you, but have you seen this? Could this possibly be causing the continuous model load issue?

<!-- gh-comment-id:4948416598 --> @Aamir3d commented on GitHub (Jul 11, 2026): cc @comfyanonymous and @Kosinkadink - FYI and not meaning to distract you, but have you seen this? Could this possibly be causing the continuous model load issue?
Author
Owner

@anr2me commented on GitHub (Jul 14, 2026):

Reference regarding FILE_FLAG_SEQUENTIAL_SCAN behavior https://devblogs.microsoft.com/oldnewthing/20120120-00/?p=8493

<!-- gh-comment-id:4971118354 --> @anr2me commented on GitHub (Jul 14, 2026): Reference regarding FILE_FLAG_SEQUENTIAL_SCAN behavior https://devblogs.microsoft.com/oldnewthing/20120120-00/?p=8493
Author
Owner

@brendanhoar commented on GitHub (Jul 14, 2026):

Reference regarding FILE_FLAG_SEQUENTIAL_SCAN behavior https://devblogs.microsoft.com/oldnewthing/20120120-00/?p=8493

Heh, yeah. It kind of pulls aside the curtain on the heuristic shit show going on at the kernel level.

I take it as a cautionary tale to be careful because of the caveats:

  1. this is what happens in win 7.
  2. this behavior is not contractual.
  3. this behavior changes from major version to major version.
  4. "memory mapped files don't go through the cache manager" might have been true with win7, but I'm not convinced that's true in win11.

So, what I get out of it is: if you specify this flag, you ought to have A/B tested all of the cases (available Ram sizes, a available VRAM sizes, model and sub model sizes, common multi-midel workflows, various storage system/filesystem arrangements, local vs network storage, etc. etc.) before committing to forcing everyone to use the nonstandard heuristics implicit (and somewhat unknowable) buried in the kernel when this flag is active.

ComfyUI allows the end user to specify enabling or disabling of memory mapping of Safetensors and other model files. What I would like to see, at a minimum, would be exposing this createfilew flag as a startup flag under user control, which would allow end users to ad hoc A/B test without a compiler environment and other developer experience needed. This would also mean the aimdo functionality would need to pass it, and the DLL calls would as well.

B

<!-- gh-comment-id:4971290538 --> @brendanhoar commented on GitHub (Jul 14, 2026): > Reference regarding FILE_FLAG_SEQUENTIAL_SCAN behavior https://devblogs.microsoft.com/oldnewthing/20120120-00/?p=8493 Heh, yeah. It kind of pulls aside the curtain on the heuristic shit show going on at the kernel level. I take it as a cautionary tale to be careful because of the caveats: 1) this is what happens in win 7. 2) this behavior is not contractual. 3) this behavior changes from major version to major version. 4) "memory mapped files don't go through the cache manager" might have been true with win7, but I'm not convinced that's true in win11. So, what I get out of it is: if you specify this flag, you ought to have A/B tested *all* of the cases (available Ram sizes, a available VRAM sizes, model and sub model sizes, common multi-midel workflows, various storage system/filesystem arrangements, local vs network storage, etc. etc.) before committing to forcing everyone to use the nonstandard heuristics implicit (and somewhat unknowable) buried in the kernel when this flag is active. ComfyUI allows the end user to specify enabling or disabling of memory mapping of Safetensors and other model files. What I would like to see, at a minimum, would be exposing this createfilew flag as a startup flag under user control, which would allow end users to ad hoc A/B test without a compiler environment and other developer experience needed. This would also mean the aimdo functionality would need to pass it, and the DLL calls would as well. B
Author
Owner

@Bit-Nibble commented on GitHub (Jul 15, 2026):

I think so too. Given the diversity of user environments, I believe providing an option to toggle this (at least for now) is a good idea—even if it is just as a sub-option of --fast.

By the way, here are the results of my testing (on Windows 11 Pro).
When running a generation in a situation where the models cannot be fully held in VRAM and RAM, the model files are naturally re-read repeatedly. However, I observed that if the cache is already full, the system does not discard old cache entries to make room; instead, it repeatedly reads from the physical disk.
What I did:

  1. Launched ComfyUI with the following startup options:

    • --fast-disk: Prefer disk-backed dynamic loading and offload over unpinned RAM.
    • --disable-pinned-memory
      I expected that models would not be held in anything other than VRAM (forcing disk-based offloading).
  2. Performed file access exceeding the available RAM capacity.
    The goal was to fill the OS file cache region (I processed numerous large files using 7-Zip's hash calculation feature).

  3. Observed physical disk reads and cache copies using PERFMON while performing two consecutive image generations.
    Results:
    Every single operation—loading each model, the step-by-step loading of offloaded parts that didn't fit in VRAM, and the model loading for the second generation—was observed to be read from the physical disk.
    Image
    Image

  4. Replaced aimdo with a version built without that flag and repeated the test.
    Result: After the initial model load, the cache was utilized, and no further reads were observed from the physical disk.
    Image
    Image

Additional Note:
Even with the flag enabled, if there was available space in the cache, the data was indeed being cached.

<!-- gh-comment-id:4977630569 --> @Bit-Nibble commented on GitHub (Jul 15, 2026): I think so too. Given the diversity of user environments, I believe providing an option to toggle this (at least for now) is a good idea—even if it is just as a sub-option of `--fast`. By the way, here are the results of my testing (on Windows 11 Pro). When running a generation in a situation where the models cannot be fully held in VRAM and RAM, the model files are naturally re-read repeatedly. However, I observed that if the cache is already full, the system does not discard old cache entries to make room; instead, it repeatedly reads from the physical disk. **What I did:** 1. **Launched ComfyUI with the following startup options:** - `--fast-disk`: Prefer disk-backed dynamic loading and offload over unpinned RAM. - `--disable-pinned-memory` I expected that models would not be held in anything other than VRAM (forcing disk-based offloading). 2. **Performed file access exceeding the available RAM capacity.** The goal was to fill the OS file cache region (I processed numerous large files using 7-Zip's hash calculation feature). 3. **Observed physical disk reads and cache copies using PERFMON while performing two consecutive image generations.** **Results:** Every single operation—loading each model, the step-by-step loading of offloaded parts that didn't fit in VRAM, and the model loading for the second generation—was observed to be read from the physical disk. <img width="720" height="470" alt="Image" src="https://github.com/user-attachments/assets/545a05e9-3643-4e5f-9944-9034b22715aa" /> <img width="512" height="961" alt="Image" src="https://github.com/user-attachments/assets/49746b98-d20c-4d8e-bb55-64e89fe1505b" /> 4. **Replaced `aimdo` with a version built without that flag and repeated the test.** **Result:** After the initial model load, the cache was utilized, and no further reads were observed from the physical disk. <img width="720" height="470" alt="Image" src="https://github.com/user-attachments/assets/211125ad-a9a7-4d8a-a948-7417e5df2a3d" /> <img width="515" height="961" alt="Image" src="https://github.com/user-attachments/assets/a4b77753-fb16-420f-9918-8e273db00401" /> **Additional Note:** Even with the flag enabled, if there was available space in the cache, the data was indeed being cached.
Author
Owner

@VladiCz commented on GitHub (Jul 15, 2026):

@brendanhoar
I have already posted here although I'm on Ubuntu and experiencing exactly the same slowdown.

I have resolved the issue with slow model reads between runs from drive/RAM to VRAM since v0.22.0 by adding --disable-pinned-memory flag. This resulted in stable and fast behavior (up to 10x speedup) I was used to before upgrading ComfyUI from v0.22.0 or maybe v0.21.0, not sure anymore.

I did try multiple flags and combination before. I now removed --fast-disk option I was testing as it did not help in my case even with dedicated nvme PCIE 3.0 drive for models/loras. I kept --high-ram as I have a RAM to spare and it was not utilized properely.

So my relevant startup flags to resolve the issue in Ubuntu are: --high-ram --disable-pinned-memory

Using pinned-memory in all systems as a default option was a bad choice as it is obviously OS or HW dependent as it can results in 10x slowdown with model reads. It can be related to the fact that I only use model + lora combination. I never use bare model without loras be it distill lora, IC-lora, other lora so the model is "constructed" dynamically from multiple lora combination.

Finally, my inference times are longer than model loading/reloding times as for past several months I was only waiting for model to be slowly read back even without changing prompt, seed etc. Now the model is also read back but in matter of seconds, not minutes.

And one more thing for NVME/SSD drive owners. The fact that your NVME/SSD drive was fast when you bought it (or on paper) does not mean it remains fast after few weeks/months/years. Some drives are know to slow down after being written and drop read/write speeds close to SATA HDD, like from 3500 MB/s to 200 MB/s. This includes read, not just writes. This is due to aging of the cells (and it is not even related to trim). The controller can often read the data correctly but with added latency making the drive much slower. "Famous" drives with this issue are Crucial P3, Crucial P3 Plus etc. I also experience this.

I write it because you might be thinking there is no issue with your NVME drive reading the models while it might be one of the factors you might be overlooking. There is plenty more however on the OS level.

<!-- gh-comment-id:4978825543 --> @VladiCz commented on GitHub (Jul 15, 2026): @brendanhoar I have already posted here although I'm on Ubuntu and experiencing exactly the same slowdown. I have resolved the issue with slow model reads between runs from drive/RAM to VRAM since v0.22.0 by adding --disable-pinned-memory flag. This resulted in stable and fast behavior (up to 10x speedup) I was used to before upgrading ComfyUI from v0.22.0 or maybe v0.21.0, not sure anymore. I did try multiple flags and combination before. I now removed --fast-disk option I was testing as it did not help in my case even with dedicated nvme PCIE 3.0 drive for models/loras. I kept --high-ram as I have a RAM to spare and it was not utilized properely. So my relevant startup flags to resolve the issue in Ubuntu are: --high-ram --disable-pinned-memory Using pinned-memory in all systems as a default option was a bad choice as it is obviously OS or HW dependent as it can results in 10x slowdown with model reads. It can be related to the fact that I only use model + lora combination. I never use bare model without loras be it distill lora, IC-lora, other lora so the model is "constructed" dynamically from multiple lora combination. Finally, my inference times are longer than model loading/reloding times as for past several months I was only waiting for model to be slowly read back even without changing prompt, seed etc. Now the model is also read back but in matter of seconds, not minutes. And one more thing for NVME/SSD drive owners. The fact that your NVME/SSD drive was fast when you bought it (or on paper) does not mean it remains fast after few weeks/months/years. Some drives are know to slow down after being written and drop read/write speeds close to SATA HDD, like from 3500 MB/s to 200 MB/s. This includes read, not just writes. This is due to aging of the cells (and it is not even related to trim). The controller can often read the data correctly but with added latency making the drive much slower. "Famous" drives with this issue are Crucial P3, Crucial P3 Plus etc. I also experience this. I write it because you might be thinking there is no issue with your NVME drive reading the models while it might be one of the factors you might be overlooking. There is plenty more however on the OS level.
Author
Owner

@brendanhoar commented on GitHub (Jul 15, 2026):

My system is windows 11 with 512GB of RAM. Models stored on a mix of encrypted nvme and external HDDs.

0.2.2 and earlier, first generations were slow due to storage I/O but subsequent were super fast, and clearly this was due to OS caching the model files. If I pre read the model files once with a tool that did large sequential reads, then as long as I stuck with all of the same models used in a complex workflow, all generations never went back to storage and all ran out of the OS cache limited by GPU only.

Now, with recent comfy, even if I manually read the model files with another tool, after the first gen, comfy goes back to the storage device again. Something is now hinting to windows 11 that the cached files can be discarded at a high priority.

It is most likely that what you are experiencing and what I am experiencing are both happening.

B

<!-- gh-comment-id:4983130342 --> @brendanhoar commented on GitHub (Jul 15, 2026): My system is windows 11 with 512GB of RAM. Models stored on a mix of encrypted nvme and external HDDs. 0.2.2 and earlier, first generations were slow due to storage I/O but subsequent were super fast, and clearly this was due to OS caching the model files. If I pre read the model files once with a tool that did large sequential reads, then as long as I stuck with all of the same models used in a complex workflow, all generations never went back to storage and all ran out of the OS cache limited by GPU only. Now, with recent comfy, even if I manually read the model files with another tool, after the first gen, comfy *goes back to the storage device again*. Something is now hinting to windows 11 that the cached files can be discarded at a high priority. It is most likely that what you are experiencing and what I am experiencing are *both* happening. B
Author
Owner

@Aamir3d commented on GitHub (Jul 15, 2026):

Also interesting that Linux systems are equally affected. I wish this was sorted as I can't use Comfy in its current state at all.

<!-- gh-comment-id:4983767767 --> @Aamir3d commented on GitHub (Jul 15, 2026): Also interesting that Linux systems are equally affected. I wish this was sorted as I can't use Comfy in its current state at all.
Author
Owner

@Bit-Nibble commented on GitHub (Jul 15, 2026):

@brendanhoar
According to the documentation for CreateFile (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew):
"Note: When CreateFile opens an existing file, it generally combines the file flags with the file attributes of the existing file, and ignores any file attributes supplied as part of dwFlagsAndAttributes."
This suggests that even for files already residing in the OS cache, the flags passed during CreateFile might be altering the cache management strategy.

<!-- gh-comment-id:4985471127 --> @Bit-Nibble commented on GitHub (Jul 15, 2026): @brendanhoar According to the documentation for CreateFile (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew): "Note: When CreateFile opens an existing file, it generally combines the file flags with the file attributes of the existing file, and ignores any file attributes supplied as part of dwFlagsAndAttributes." This suggests that even for files already residing in the OS cache, the flags passed during CreateFile might be altering the cache management strategy.
Author
Owner

@VladiCz commented on GitHub (Jul 16, 2026):

Also interesting that Linux systems are equally affected. I wish this was sorted as I can't use Comfy in its current state at all.

I do not think the issues are equal for Linux and Windows as there are multiple factors, some are relevant for many OS's and some are specific to OS. So there will be no solution fits all. At least I was able to solve the issue with the startup flag, which does not seem to be the case for Windows users.

Also there is the issue of people on Windows who actually run ComfyUI in WSL/WSL2. This brings completely different sum of issues which are related to both Windows and Linux running in container, which has a lot of downsides regarding accessing the resources. I migrated from WSL2 to pure Linux as WSL2 is not a good fit for ComfyUI, maybe from the start, but not in the long run and more heavy use. So for those who are using WSL2 there is really only solution to migrate ComfyUI directly to the host system, preferably Linux.

I really recommend people to dual boot. I started with dual boot and so far not booted back to Windows in 6 months. It remains installed but I do everything on Linux now. Fedora, Ubuntu are a good choice btw.

<!-- gh-comment-id:4989424688 --> @VladiCz commented on GitHub (Jul 16, 2026): > Also interesting that Linux systems are equally affected. I wish this was sorted as I can't use Comfy in its current state at all. I do not think the issues are equal for Linux and Windows as there are multiple factors, some are relevant for many OS's and some are specific to OS. So there will be no solution fits all. At least I was able to solve the issue with the startup flag, which does not seem to be the case for Windows users. Also there is the issue of people on Windows who actually run ComfyUI in WSL/WSL2. This brings completely different sum of issues which are related to both Windows and Linux running in container, which has a lot of downsides regarding accessing the resources. I migrated from WSL2 to pure Linux as WSL2 is not a good fit for ComfyUI, maybe from the start, but not in the long run and more heavy use. So for those who are using WSL2 there is really only solution to migrate ComfyUI directly to the host system, preferably Linux. I really recommend people to dual boot. I started with dual boot and so far not booted back to Windows in 6 months. It remains installed but I do everything on Linux now. Fedora, Ubuntu are a good choice btw.
Author
Owner

@kingp0dd commented on GitHub (Jul 16, 2026):

I experience the same, I'm on Linux

https://github.com/Comfy-Org/ComfyUI/issues/14705

<!-- gh-comment-id:4991382872 --> @kingp0dd commented on GitHub (Jul 16, 2026): I experience the same, I'm on Linux https://github.com/Comfy-Org/ComfyUI/issues/14705
Author
Owner

@rattus128 commented on GitHub (Jul 22, 2026):

https://github.com/Comfy-Org/comfy-aimdo/pull/76

<!-- gh-comment-id:5041467474 --> @rattus128 commented on GitHub (Jul 22, 2026): https://github.com/Comfy-Org/comfy-aimdo/pull/76
Author
Owner

@brendanhoar commented on GitHub (Jul 30, 2026):

were any wheels/libs built for #76 so that this can be tested in the field?

<!-- gh-comment-id:5133160317 --> @brendanhoar commented on GitHub (Jul 30, 2026): were any wheels/libs built for #76 so that this can be tested in the field?
Author
Owner

@anr2me commented on GitHub (Jul 31, 2026):

were any wheels/libs built for #76 so that this can be tested in the field?

  1. Open the drop down button of "All checks have passed"
  2. Click one of the "Build Wheels" link and Open the drop down button on that page.
  3. Click "Summary", at the bottom of the Summary page you will find a few of wheel artifacts you can test.
<!-- gh-comment-id:5139095623 --> @anr2me commented on GitHub (Jul 31, 2026): > were any wheels/libs built for [#76](https://github.com/Comfy-Org/comfy-aimdo/pull/76) so that this can be tested in the field? 1. Open the drop down button of "All checks have passed" 2. Click one of the "Build Wheels" link and Open the drop down button on that page. 3. Click "Summary", at the bottom of the Summary page you will find a few of wheel artifacts you can test.
Author
Owner

@brendanhoar commented on GitHub (Jul 31, 2026):

were any wheels/libs built for #76 so that this can be tested in the field?

1. Open the drop down button of "All checks have passed"

2. Click one of the "Build Wheels" link and Open the drop down button on that page.

3. Click "Summary", at the bottom of the Summary page you will find a few of wheel artifacts you can test.

Thanks, unfortunately after doing the embedded\python.exe pip -m install 'path\to\whl'
I get the following on startup:

15:15:19.551 [Debug] [Load ComfyUI Self-Starting #0] All tasks done.
15:15:19.552 [Debug] [Load ComfyUI Self-Starting #0] Will validate required libs...
15:15:19.560 [Error] Final error (4) while initializing backend #0 - ComfyUI Self-Starting, giving up: System.FormatException: The input string 'dev12' was not in a correct format.
   at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
   at System.Version.TryParseComponent(ReadOnlySpan`1 component, String componentName, Boolean throwOnFailure, Int32& parsedComponent)
   at System.Version.ParseVersion(ReadOnlySpan`1 input, Boolean throwOnFailure)
   at System.Version.Parse(String input)
   at SwarmUI.Builtin_ComfyUIBackend.ComfyUISelfStartBackend.DoLibFixes(Boolean doFixFrontend, Boolean doLatestFrontend) in G:\___all_webuis\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUISelfStartBackend.cs:line 608
   at SwarmUI.Builtin_ComfyUIBackend.ComfyUISelfStartBackend.Init() in G:\___all_webuis\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUISelfStartBackend.cs:line 474
   at SwarmUI.Backends.BackendHandler.LoadBackendDirect(BackendData data) in G:\___all_webuis\SwarmUI\src\Backends\BackendHandler.cs:line 606

<!-- gh-comment-id:5147045107 --> @brendanhoar commented on GitHub (Jul 31, 2026): > > were any wheels/libs built for [#76](https://github.com/Comfy-Org/comfy-aimdo/pull/76) so that this can be tested in the field? > > 1. Open the drop down button of "All checks have passed" > > 2. Click one of the "Build Wheels" link and Open the drop down button on that page. > > 3. Click "Summary", at the bottom of the Summary page you will find a few of wheel artifacts you can test. Thanks, unfortunately after doing the embedded\python.exe pip -m install 'path\to\whl' I get the following on startup: ``` 15:15:19.551 [Debug] [Load ComfyUI Self-Starting #0] All tasks done. 15:15:19.552 [Debug] [Load ComfyUI Self-Starting #0] Will validate required libs... 15:15:19.560 [Error] Final error (4) while initializing backend #0 - ComfyUI Self-Starting, giving up: System.FormatException: The input string 'dev12' was not in a correct format. at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value) at System.Version.TryParseComponent(ReadOnlySpan`1 component, String componentName, Boolean throwOnFailure, Int32& parsedComponent) at System.Version.ParseVersion(ReadOnlySpan`1 input, Boolean throwOnFailure) at System.Version.Parse(String input) at SwarmUI.Builtin_ComfyUIBackend.ComfyUISelfStartBackend.DoLibFixes(Boolean doFixFrontend, Boolean doLatestFrontend) in G:\___all_webuis\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUISelfStartBackend.cs:line 608 at SwarmUI.Builtin_ComfyUIBackend.ComfyUISelfStartBackend.Init() in G:\___all_webuis\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUISelfStartBackend.cs:line 474 at SwarmUI.Backends.BackendHandler.LoadBackendDirect(BackendData data) in G:\___all_webuis\SwarmUI\src\Backends\BackendHandler.cs:line 606 ```
Author
Owner

@anr2me commented on GitHub (Aug 1, 2026):

Looks like it's because artifacts are usually using text letters in the version (ie. dev) instead of just numbers. While Version.Parse in SwarmUI's ComfyUISelfStartBackend.cs cannot process text letters like dev.

So the version handling probably need to be fixed first in order to support this kind of versioning.

<!-- gh-comment-id:5148627114 --> @anr2me commented on GitHub (Aug 1, 2026): Looks like it's because artifacts are usually using text letters in the version (ie. `dev`) instead of just numbers. While `Version.Parse` in SwarmUI's `ComfyUISelfStartBackend.cs` cannot process text letters like `dev`. So the version handling probably need to be fixed first in order to support this kind of versioning.
Author
Owner

@brendanhoar commented on GitHub (Aug 1, 2026):

I'll retest a bit later, mcmonkey said it's only during autoupdate, which can be disabled for this purpose.

<!-- gh-comment-id:5151517217 --> @brendanhoar commented on GitHub (Aug 1, 2026): I'll retest a bit later, mcmonkey said it's only during autoupdate, which can be disabled for this purpose.
Author
Owner

@anr2me commented on GitHub (Aug 1, 2026):

Besides dev they might want to handle post too while at it 😅 because the text "post" often used in many nightly wheel version.

<!-- gh-comment-id:5152131922 --> @anr2me commented on GitHub (Aug 1, 2026): Besides `dev` they might want to handle `post` too while at it 😅 because the text "post" often used in many nightly wheel version.
Author
Owner

@brendanhoar commented on GitHub (Aug 30, 2026):

@rattus128 -

This appears to be resolved from my perspective, I forgot to post here when it occurred.

I saw that you'd merged the recommended change into comfy-aimdo which then flowed to comfyui. At which point the issue fully resolved for me.

I can't say for SURE whether this recommendation was THE fix, but it was one of three commits in aimdo that went out that day, and the problem went away that day, so...probably?

Anyway, thanks for addressing it!
B

<!-- gh-comment-id:5471210597 --> @brendanhoar commented on GitHub (Aug 30, 2026): @rattus128 - This appears to be resolved from my perspective, I forgot to post here when it occurred. I saw that you'd merged the recommended change into comfy-aimdo which then flowed to comfyui. At which point the issue fully resolved for me. I can't say for SURE whether this recommendation was THE fix, but it was one of three commits in aimdo that went out that day, and the problem went away that day, so...probably? Anyway, thanks for addressing it! B
Sign in to join this conversation.
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Comfy-Org/comfy-aimdo#22
No description provided.