mirror of
https://github.com/Comfy-Org/ComfyUI-Manager.git
synced 2026-11-05 07:47:25 +00:00
[GH-ISSUE #164] Socket Closed, Exception on Fetch Model Updates, followed by hang when going to "Install Models" since a version early this morning #87
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/ComfyUI-Manager#87
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @NeedsMoar on GitHub (Nov 5, 2023).
Original GitHub issue: https://github.com/Comfy-Org/ComfyUI-Manager/issues/164
LOG
This is what I get when the usual fetch node updates finishes(?); or at least it appears to have gone through all installed nodes at this point. It takes maybe 3x longer than it used to, which was already a long time.
After this attempting to "install custom nodes" just hangs on the update check, CTRL-C will no longer close comfy at that point. Unfortunately I'm not even getting an appcrash event in windows event log over the force-close on the terminal. No memory access or network errors either.
I manually updated the manager itself just now (git pull) but it's still happening. Unfortunately that log output is practically useless for troubleshooting so I'm not sure what else I can give you to help track this one down.
@NeedsMoar commented on GitHub (Nov 5, 2023):
"Update All" throws the same error at the end, but seems to successfully update all of the nodes. CTRL-C still works after that reset connection error as long as nothing else is done. On restart it looks like only impact-pack updated dependencies (I think it does this itself) instead of all updated nodes as usual so I'm guessing the startup script to re-run the requirements wasn't created which would mean either that code changed or the disconnect happens between printing "Update done." and whatever was supposed to be creating the startup scripts? Maybe that helps a little more.
Update done.
ERROR:asyncio:Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
@ltdrdata commented on GitHub (Nov 5, 2023):
Try rolling back to a previous commit and see if the issue is resolved.
git checkout
74e1f624@NeedsMoar commented on GitHub (Nov 5, 2023):
I went back further to just before the snapshot stuff to
2d6633esince that was where it worked for me previously but also because I'm kinda an idiot with git and when checkout didn't remove the snapshots folder I assumed it didn't work, dug around online and found that:Apologies for the barely-rant rant.
Anyway,
2d6633ecan fetch the nodes successfully (except ComfyUI-Manager because of the detached head status of course) and didn't throw the connection error after doing the fetch nodes, and I'm able to go into install nodes and update. I usually do it that way even though it's way slower because new nodes / features get added so fast that I like to scan the list and check for large changes on the various git repos and newly duplicated functionality (the 27 different SDXL prompters and resolution select nodes come to mind).Startup scripts go through and run the requirements again as before too.
Notably the speed regression was gone and fetching the custom nodes takes maybe 5 minutes less than on head. Dunno why, I couldn't see any changes that should have caused that issue.
After I update everything again I'm going to try out
94cb2f8since I kinda suspect the snapshot stuff is doing this somehow and the error is coming from one of those endpoints. I also realized I had --dont-print-server specified because of the web server console spam bug from a while back so that's not specified now and maybe I'll get more info about exactly what page was messed up either on94cb2f8or HEAD.@NeedsMoar commented on GitHub (Nov 5, 2023):
94cb2f8fails at the end of fetch nodes with the same error as the original post. Since I had just updated them I'm pretty sure there wasn't even anything new. I turned access logging back on in server.py (forgot about that part) and am running again to see if I get a URL out of it before the exception.@NeedsMoar commented on GitHub (Nov 5, 2023):
I disabled privacybadger and ublock origin and didn't get errors, but this was the first time no updated nodes were found so I'd take that with a grain of salt. On my last attempt before that I could update the nodes after the fetch but it took roughly 5 minutes after node fetch completed before I could enter the install list when clicking on it, and the same between installing each node so there's something messed up with performance but I don't know what and I can't say that it's actually related to the manager. In any case I got the current release updating nodes and itself and comfy properly even if it takes longer so I'm going to mark this as closed unless I find something that otherwise indicates that manager was the real cause. The interaction of so many nodes in comfy is sort of a recipe for things like this from time to time.
I noticed that the websocket attempts were returning 101 request different service. In firefox developer console, the local http requests for websockets were being switched to https which I'm guessing the local server doesn't support (or the encryption methods weren't supported). I have forced https / require https disabled because of the various web interfaces I can't be bothered to generate valid certs for like the hardware RAID controller on my old machine, the BMC on this one (during the 5 minutes I actually connect it when a BIOS update is released), and a Xerox laser printer. Fallback to https is still turned on though, and I'm wondering if the UI spamming websocket requests while the server is busy eventually tries the fallback and fails because of some aiohttp config or other firefox security settings. Something had changed recently that was causing CORS errors on some web merchants so that last item wouldn't be out of the question.
@NeedsMoar commented on GitHub (Nov 5, 2023):
Here's what the logs looked like on the working run with 2 updated nodes just so you havve a reference because I'll lose this:
@ltdrdata commented on GitHub (Nov 6, 2023):
Thank you for the detailed report.
@NeedsMoar commented on GitHub (Nov 6, 2023):
No problem, I'd be lost trying to keep track of Comfy's custom nodes without your extension. If I get bored enough and it starts acting up again I'll attach a debugger to python and see if I can get more information on the parameters, but from what I know of the parameter boxing and ref counting stuff that goes on in C / C++ code that python uses it might be more of a PITA than I want to deal with, and I'm in a quagmire of trying to build a pytorch-2.2.0-cu123 on Windows right now to see if it makes a difference since they added lazy loading; it's just not clear whether that enhancement to the runtime can apply to anything built against 12.1 or if it needs rebuilds.
I tried one more update earlier and there was no crash when doing "update all" with ~4 extensions updated. I also didn't see any more of those odd websocket calls in the logs. I'll re-open this if it happens again and it seems to be related.