mirror of https://github.com/Comfy-Org/comfy-qa-tools.git synced 2026-11-04 23:47:20 +00:00

QA tooling for testing Comfy

Python 100%

Find a file

aliranjah 024b99f9a7 1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 ) * discover no longer crashes on a mix of reserved and unreserved boxes A reserved row in discover's table has a sixth cell and every other row has five, and the table padding measured every row as long as the first. So any mix of the two crashed with an IndexError, in either order, after printing its heading and before writing anything (O22). Rows may now be different lengths, and a short one simply ends early. The test adopts a reserved box and a box with no GPU together, in both orders: both are written, and only the reserved one says it is reserved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Every printed reset-windows-password command carries --quiet The command printed when a Windows password reset answers with no credentials went out without --quiet, so pasted into a script it stopped to ask and exited 1 (O25). Its three siblings already had the flag. The guard that should have caught it read two files by name and matched the command as one literal, and this one was in a third file, split across two literals. The guard now walks the syntax tree of every module in comfy_qa/, where adjacent literals are already one string, and checks every printed gcloud delete and reset-windows-password for --quiet. tests/printed.py holds that walk so the next guard over printed lines can use it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A box named stockout is not a stock-out, and one whitespace fold for all readers The bare word "stockout" was one of the stock-out signs, and Google echoes the resource path in most of what it refuses, so an already-exists or not-found about a box called qatest-stockout read as "the zone is out of stock, try another" (O25). The sign is now Google's own state code, state:STOCKOUT, which a box name cannot hold. The real line-wrapped text from the live pass, and the state code on its own, are still recognised. gcloud wraps its YAML errors anywhere, and only the stock-out check folded whitespace before matching. classify and suggested_zones now go through the same fold, gcloud.folded, so a wrap can no longer hide Google's suggested zone or turn a missing project into an unknown failure. The test breaks each real gcloud text the suite holds at every space in turn and checks all three readers give the same answer as on the unbroken text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A --region stock-out tells the truth on a one-region project, and keeps the box Three follow-ups to the live pass's --region fix (O23): When the region --region named is the only one the project can use the card in, the stock-out said "not everywhere this project can use the card" and offered "drop --region", which retried the same zones. It now says that is the only region there is and that there is nowhere left to try. Whether there is anywhere else comes from the catalogue create already reads, so no call is added. The "same box" in that remedy dropped --disk, and --name for a box that was not reserved, so pasted back it made a 200 GB box under another name. It now carries the name for every box and the disk whenever it is not the default. The untried-regions remedy beside it had the same gap and now uses the same helper. The no-GPU half of the wiring that tells the stock-out a region was named had no test; one drives create --gpu none --region through the CLI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * comfy-qat keeps its own Google project, in the host list The project was gcloud's: setup --project ran gcloud config set project, and every command asked gcloud config get-value project. So comfy-qat and every other gcloud user on the machine shared one setting, and moving comfy-qat to a new project moved everything else with it. Now one project = "<id>" line at the top of the host list says it. setup --project writes it through the same checked rewrite as every other change to that file, and nothing in the tool can set gcloud's project any more. create, discover, quota, status and down --all read the line; without one they use gcloud's current project as before, and say so. Every gcloud call that acts on a project carries --project, so the line reaches Google whatever gcloud is set to. A box keeps the gce_project its own entry names, so after a switch the old boxes are still stopped, started and deleted where they are. list says which project each box is on and where a new one will go. The docs say how to switch and that there is no move between projects. A misspelt line, or a display name where the ID goes, is refused rather than ignored, since ignoring it would send every call to gcloud's project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Every command the tool prints runs as pasted Two things stopped a printed command working when pasted whole (O25): A printed comfy-qat delete <name> stopped to ask for the box's name, so from a script or an agent it exited 2 with nothing deleted, and for a reserved box that is the command that stops the bill. Every printed delete now carries --yes. Typing delete yourself still asks. Offered commands carried their reason as a trailing # note, as in "comfy-qat down comfy-linux # stop the box, stop paying". An interactive zsh without interactivecomments, which is how zsh starts, passes the # and every word after it to the command, which refuses them. The reason is now a line of its own ending in a colon, with the command alone under it. say.offer builds that shape, and a test reads every string in the package and fails on a command line with a # note on the end. The docs that quote these lines are updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.3.0 notes say what the live run really found The 1.3.0 changelog said the live run found four defects, fixed below. It found ten, and four were fixed; the entry now says so and names the other six. It had no line for the fix to how a stock-out is recognised in Google's line-wrapped text, and now has one. The acceptance pack still said what ComfyUI calls the device on a box with no GPU had not been seen; the live run recorded it, and the pack now quotes the stamp line it printed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Release 1.3.1 Version 1.3.1, with a changelog entry naming each fix in plain words and saying what ran on real Google boxes and what did not. The README's Status line names the version, which the suite requires; nothing else in it moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Every reader of gcloud's text goes through the one fold `gcloud.folded`'s docstring says every reader that matches a phrase in what gcloud printed goes through it. Three did not: `is_already_asked`, `_missing_resource` and `is_auth_failure`'s text fallback matched the raw text, so a wrap gave False, "" and False. Each failed in the safe direction, but the claim was false. All three fold now, phrases too, and the wrap test reads them with not-found and quota-preference texts added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A --zone stock-out offers the same box, and two guards read literal names After a `--zone` stock-out a box that was not reserved was offered a bare `comfy-qat create`: no `--os`, `--gpu`, `--name` or `--disk`, so pasted with no terminal it exited 2. Both branches use `_same_box` now, and a CLI test compares the offered command with the arguments and pastes it back. The guards for a printed gcloud delete or reset without `--quiet`, and a printed `comfy-qat delete` without `--yes`, matched only an f-string hole after the verb, so a box name typed out passed. They match any word now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * setup --help says --project replaces the host list's project line The help said an existing host list is "read, never replaced" and only appended to, and `--project` said "use this project instead of asking". Neither said `--project` writes the project line, over any already there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs quote the endings 1.3.1 prints Acceptance checks V4 and V6 and two troubleshooting entries still quoted commands with a `# note` on the end and a printed delete without `--yes`, so a tester following them would fail correct output. They quote the reason line, the command alone under it, and `delete --yes` now. The 1.3.1 entry names the `--zone` remedy and the three readers folded since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The tunnel's readers fold gcloud's text, and lifecycle prints one command a line tunnel.py's three readers of a dead tunnel's log (_iap_refused, still_booting, _credentials_expired) lowered the text and nothing more, so a log broken at `port⏎22` read a box still booting as one with nothing listening, and `go` told it to start ComfyUI first. They go through gcloud.folded now, as every other reader does, and the wrap test reads them: the real booting log, the 4003 refusal and an expired session, plus every phrase each one looks for broken at each of its spaces (round-1 re-check of F7). lifecycle's reserved-box fix was one sentence with two commands in it, `stop the box: comfy-qat down X — it is reserved, so the bill only stops with: …`; each command is alone on its line under its reason now, and the one fix that joined two commands with `&&` prints them on two lines. The quota-preference fixture in the wrap test is word for word what the suite records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * No writer breaks a command it is handed `say.wrapped` wrapped whatever it was handed, and two commands came out of it in two pieces once their values were in. `move`'s note about a slower boot disk goes through `warn`, and the `gcloud compute disks delete … --project=… --quiet` in it was broken at 96 columns, so the first line, copied, deletes on gcloud's own project and stops to ask. And `move`'s leftovers report handed `detail` each delete indented two spaces, which `_is_prose` read as prose and wrapped the same way (round-1 F7, the move leftovers block). A line that is a command is left whole at its indent now, and a command a sentence ends on after its colon is one word to textwrap, so it moves to the next line in one piece. `detail` asks `_is_prose` of the text, not its indent. Held for every line in the package that offers a command, at three value lengths and with the width falling everywhere along it, through each writer's prefix; for the move note and the leftovers report on the printed output; and for a sentence pushed a word at a time across the width. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Every printed command is the command alone, on its project, for the same box Round two of O25e, O23b and S1-11b, each fixed for the class the audit found rather than the line it quoted. A printed command carries nothing but the command. The notes that rode on command lines are on lines of their own above them, ending in the colon that introduces the command: the `# note` fragments `region_problem` and `region_shape_problem` hand back, which `setup`, `create` and `quota` spliced into whole commands (`auth.spliced` puts the command on the flag lines only); the `--allow-lower # …` remedies; the em-dash notes after commands in create, quota and gcloud's DENIED advice; `, then wait for Google`; `… --gpu t4, the same n1-standard-8 machine …`; setup's sign-off table (`comfy-qat list see them`), whose `stamp` line is now printed only for a local machine the host list declares, by its name; `quota request ... --dry-run`, which printed a literal `...`; the project-fallback note, whose command a long project ID pushed across the wrap; and a project-ID refusal that ended on a command and is put inside other sentences. Every printed gcloud command that acts on a project names it. `down --all`'s stop lines for boxes nobody declared, the ceiling refusal's stop line and listing, the zones/regions/machine-types listings, the release and delete lines, and `prune`'s "find where each one really is", which named `gcloud compute instances list` mid-sentence and now offers it on the entry's project. With no project in hand the flag reads `--project=<your-project-id>` (`reservation.NO_PROJECT`) rather than being left out, which would act on gcloud's own project; the stale comment that gcloud's project is the one this tool follows is gone. Every "the same box, somewhere else" is the box that was asked for: `_refuse_full`, `_where_the_leftover_is` and the capped plain stock-out pair use `_same_box`, and host.py's `--zone`/`--region` conflict and region-typo rewrites keep `--name` and `--disk` (`create.besides_where`). One CLI test compares each offered command with the refused create's arguments and pastes it back with --dry-run and no terminal. The guards, in the same commit because the old ones would read the new output as wrong and the new ones the old: The first S1-10 guard read one shape — two spaces and a `#` after a line starting `comfy-qat `, `gcloud ` or a hole — and the rest of the class went on printing. The question is asked of the command now (`tests/commandlines.py`): a `#` with any space before it, a dash, a run of spaces, or a word that ends a phrase, after the program, is something other than the command; an `ssh` line is a command like the others. Two readings, because neither reaches everything. `test_host_costs` walks every string in the package (a value is `{}` there). `conftest` records every line every test prints and holds it to the same rule with the real values in, parses `comfy-qat` lines with the CLI's own parser so a word it would refuse is caught whatever it looks like, and holds a printed gcloud line to its `--project` and to `--quiet` on the same physical line. Two `test_say` tests that hand `write_fix` notes of their own are marked and listed. S1-11b: a guard that every printed gcloud compute/quotas/services/billing command, offered or named in passing, carries `--project` (only `billing accounts` is exempt, by name), and CLI tests where the host list's project differs from gcloud's: `down --all`'s and the ceiling refusal's stop lines, pasted, reach the host list's project. F6: every argv the real `Gcloud.run`/`run_interactive` is handed, and every fake standing in for `run`, is refused when it acts on a project without naming it, suite-wide — so an argv built with `str.split` on a command no test drives is caught too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.3.1 entry says what round two fixed and what ran live, on which commit S1-1 was run live again on `acf517c`, the last commit before this one, from a host list with no local machine in it and ports above 8190: 1.3.0 crashed on a reserved and an unreserved box with no GPU, this adopted both, and the printed `down`/`delete --yes` removed both with no terminal; the four lists were empty before and after. The entry names that commit, and says what each round-two fix is in plain words: notes off command lines, no command broken by the wrap, `--project` on every printed gcloud command, the same box wherever "somewhere else" is offered, and the tunnel's readers folded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * No test asks the developer's own ComfyUI, and the tripwire says so when one does Three tests ran `list --live` against a `[hosts.local] port = 8188` entry, which made six real GET /system_stats per full run to whatever ComfyUI was running on 127.0.0.1:8188, passing either way. They now stub `host._answering`, as test_switch and test_docs already did. The network tripwire refuses loopback on 8188-8190 too, and keeps every refusal for a teardown check: `CliRunner` turns a raised refusal into exit code 1, and one of the three accepts exit 1, so raising alone was not seen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * down --all reads every project in play for boxes nobody declared After a project switch, with an old box on the old project and new work on the host list's own, `down --all` listed machines only on the first declared box's project. A box running on the new project in no entry went unread, and the run ended on "Nothing is now." while it billed. Reservations were already read on both. `_undeclared_and_running` now reads every project a declared box is on plus the host list's, as `_reservations_on` does, and returns None if any one read fails. A declared box is matched on its own project, so the same name on another project is still reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * setup says what an adopted reserved box bills, as discover does `setup`'s discovery pass wrote `gce_reservation` for a reserved box and then said only "added <name> — …, running, port …". `discover` says the box bills every hour, running or stopped, until it is deleted, and prints what ends that. setup now prints the same ending from the same functions: `rsv.bill`, then `reserved_stop_lines` for a running box or `rsv.stop_line` for a stopped one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs say down --all reads the project the host list names Three passages said that with no cloud box declared `down --all` reads the project gcloud is pointed at, and that this is where a half-finished `create --reserve` leaves its reservation. With 1.3.1's project line it reads the line's project, beside every declared box's, and `create` makes the box there too; gcloud's is read only with no cloud box and no line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.3.1 entry names this round's fixes and no longer calls state:STOCKOUT Google's `state:STOCKOUT` is in no real Google refusal kept on record; the form comes from the test written with the first stock-out fix (#17). The CHANGELOG and the comment beside the sign say so instead of calling it Google's own code. The entry also names the two fixes to `down --all` after a switch and to `setup` adopting a reserved box, says the suite no longer reaches a ComfyUI on the machine, and says the commits after the live run's ``acf517c`` change none of the code that run used, where it called ``acf517c`` the last code commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * No comment calls state:STOCKOUT Google's own state code C2 from the step 1 audit (behaviour:F4). The comment in test_lifecycle.py now says what lifecycle._CAPACITY_SIGNS says: the code is kept because the first stock-out fix's test has it, and no real refusal on record carries it. A test holds every file under comfy_qa, tests and docs to that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * down --all reads gcloud's project whenever the host list names none C1 from the step 1 audit (behaviour:F2). With a box declared, no project line and gcloud pointed at another project, create makes boxes on gcloud's project, and down --all read only the declared box's. Both of its project reads now go through one helper that adds gcloud's project whenever the host list names none, and a gcloud that cannot name it withholds the all-clear. cost.md and troubleshooting.md say exactly that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The money tests' fake clouds answer gcloud's project down --all now asks gcloud's project whenever the host list names none (previous commit), and these two fakes refuse any call they were not told about. They answer the project their declared boxes are on, so every sentence they hold is read off the same projects as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * comfy-cli installs the chosen build on the box Choosing a build is now a module of its own (comfy_qa/build.py): a ComfyUI release, PR or commit, a frontend release or PR, Manager (on by default), and a torch pin, refused before anything else when two backends or two frontends are given, and checked read-only against GitHub so a tag, PR or commit that does not exist is refused at no cost. The install on the box is comfy-cli's own: a pinned comfy-cli 1.24.0 in a venv of its own, then comfy install with the build's options, printed before it runs, under DO_NOT_TRACK so it does not pay its five-second exit tax. A pin on torch reaches every pip install comfy-cli runs as PIP_CONSTRAINT. A build change on a box keeps the checkout and its nodes and models (comfy install --restore); only a clean install deletes it. The box records the build it was asked for, and a facts command reads back its commit, branch and record. The suite refuses any test that would ask the real GitHub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * create, go and switch choose the build; stamp shows the one running create, go and switch take --version/--pr/--commit, --frontend-version or --frontend-pr, --manager/--no-manager and --torch; go and switch take --clean. Each is checked before Google is asked anything, and each says which build it chose, the default included. create writes the build into the box's host list entry, and the first go installs it. A box that already has ComfyUI is given a new build in place, keeping its custom nodes and models; a frontend release alone is a launch option, not an install. Ready means the build asked for: once ComfyUI answers, its version, the frontend it serves, whether Manager answers, torch and the checkout's commit are read off the box, and a contradiction is said plainly with the command that fixes it, with nothing opened and nothing reported ready. stamp reads the build off the running machine, never off what was asked for: the ComfyUI version and the checkout's commit and PR, the frontend actually served, Manager on or off, and torch. A Windows GPU box installs its NVIDIA driver itself, with the method Google now documents (sysprep-specialize-script-url pointed at its own script), and go waits for the driver on Windows as it does on Linux. No PowerShell step is printed. The Linux path is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Every refusal of a build, and the Windows driver, have their entries troubleshooting.md gains 'Choosing the build', with an entry for every way a build is refused before anything exists and for the build that came up wrong, and its NVIDIA driver section says Windows installs the driver itself with the method Google now documents, and where its log is when it does not. commands.md names the build options on create, go and switch and what stamp now shows; hosts.md documents the build field; the known-fields quotes and the create dry-run sample say what the tool prints. The docs walk counts BuildError's messages, floors build.py, and raises the page's two floors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * IAP's 'Failed to lookup instance' on a box just made is a box still starting The first go on both boxes the step 2 live run created, Linux and Windows, was refused by IAP with 4047 'Failed to lookup instance' about a minute after create returned, and go reported a failed tunnel. Google had answered RUNNING a second before, and the same command a minute later worked. It is read as still booting now, so go waits it out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Changing a running box's build says whose ComfyUI it stopped It said 'stopped the ComfyUI this run started' about a ComfyUI an earlier go had started (live-s2 evidence/07). stop_running now has its own sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs say how to choose the build, and that Windows installs its driver machines.md gains 'Choosing the build' — the four parts, what each is when left off, comfy-cli doing the install, bad input refused at no cost, a build change keeping nodes and models, ready meaning the build asked for — and 'Pinning PyTorch', which quotes Miles's recommendation as written and says the wheels exist. The README says the same in brief, and both say a Windows box now installs its NVIDIA driver itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A comfy-cli install or build change that did not finish says where to look The install entry says what the comfy-cli install prints and what each INSTALL_INCOMPLETE line means, and that running it again resumes; a build change that failed, and a frontend-release change that could not be recorded, have entries of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A Windows box is made with the SSH server Google says it needs enable-windows-ssh=TRUE alone installs no SSH server. Google's own command for a Windows VM reachable over SSH also sets sysprep-specialize-script-cmd to 'googet -noconfirm=true install google-compute-engine-ssh', and the step 2 live run found a box made with the key alone never listening on 22, so go waited out its 300s. create now sets both on every Windows box. The specialize -cmd runs before the driver's -url on first boot, in the order Google documents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Release 1.4.0 Version 1.4.0, with a CHANGELOG entry that says what ran on real boxes and on which commit, and what did not: the build choice ran on a Linux box; the Windows half did not, and the entry says why. The README's status says the same, and the stamp lines in machines.md are the ones the live run printed. No module restates the version in a comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A Windows box gets Git and Python from their publishers, not winget Windows Server 2022 has no winget, so `go` on a real Windows box stopped at "comfy-cli did not install" (live-s2 evidence/41). Git 2.56.0.4 and Python 3.12.10 now come from git-for-windows' and python.org's own installers, each refused unless its SHA-256 is the pinned one, run silently with Start-Process -Wait -PassThru, and judged by exit code and by the file it leaves. The venv is made with that Python by path, and both Windows launches put Git's directory on PATH for Manager. A guard holds every Windows command under cmd.exe's 8191-character line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs and the 1.4.0 entry say where Windows gets Git and Python Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A Windows install says it installs Git and Python, not apt The line above the box's own install log said "apt, git, then pip" on every box; the first live Windows install that got that far printed it above a log with no apt in it. Each operating system now names its own steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A Windows box gets the Visual C++ runtime, and ComfyUI outlives its SSH call Two defects older than this release, found on the live run once Git and Python were in place (live-s2 evidence/44-50): - torch died at import with "Microsoft Visual C++ Redistributable is not installed" and WinError 126 on c10.dll. The install now fetches it from Microsoft's permalink, refuses it unless Microsoft signed it, runs it with /install /quiet /norestart and checks the registry key Microsoft says records it, skipping it where the key is there. - The detached launch used Start-Process, and a process started that way from an SSH session did not reach its first line once the session had returned; the same body started through WMI ran to its end. The launch goes through Win32_Process.Create, says STARTED only on ReturnValue 0, and puts Git on that process's own PATH. Each installer's URL, file and target are written once, which keeps the widest Windows install at 7,244 characters, under cmd.exe's 8,191. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs and the 1.4.0 entry say what a Windows box now installs and how it launches Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.4.0 entry says what ran live, on which commit, and what did not Windows with no GPU ran end to end on `ecb3557` after three fixes found on the way; the one GPU run found no L4 free in six zones and made nothing; nothing named qatest- is left on the project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The README says Windows ran live; the launch finding names the box it was seen on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A wrong-build fix names the build, Manager included, and the echoed comfy command is the one that runs From the step 2b audit (verify-s2b-code, verify-s2b-behaviour): - code:F2 / behaviour:F2. For the default build the fix printed when the running ComfyUI contradicted it was a bare `comfy-qat go <box>`, and a go that names no build onto a serving box asks it nothing: exit 0, browser opened, Manager still off. The fix now names the whole build with Manager (`Build.flags(explicit_manager=True)`), and `go --manager --new-window` hands `--manager` on. Tests paste the printed fix: it puts Manager back, and on a box still wrong it is refused again. - code:F4 / behaviour:F10. The `+ comfy ...` echo said `comfy` and wrote `#17497` bare. Linux echoes /opt/comfy-qat/venv/bin/comfy with the same quoting as the command; Windows echoes comfy.exe by path and quotes a `#`-led item. A test hands both Linux lines to a real sh and compares argv. - behaviour:F9. `go --clean` on a box with ComfyUI said "ComfyUI is not there"; it now says it is removing it and installing from nothing. - code:F5. Tests for the eight planted defects that survived: a frontend PR built on the box (Linux and Windows) and served from FRONTEND_ROOT, stamp's PR and commit from git and not the record, and the torch, commit and PR contradictions. - code:P5. The "a part that cannot be read does not block" rule has tests. - code:P2. Build refusals name the option in a sentence, never on a line of its own that reads as a command. - code:P1. The GitHub tripwire keeps every refusal and fails the test at teardown, as the network tripwire does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs say what 1.4.0 does about the Windows driver and the wrong-build fix, and the 1.4.0 entry what ran From the step 2b audit: - behaviour:F5. getting-started.md still said a Windows box needs a person at a PowerShell prompt for its driver, and test-criteria.md K5b failed any automated Windows recipe, so 1.4.0 would fail its own pack. Both now say what 1.4.0 does: Google's script as sysprep-specialize-script-url, `go` waiting for nvidia-smi, the serial console when it does not come up. K5b checks the plan line and the metadata; K5b2 the serial-console fix. - behaviour:F6. The 1.4.0 entry said the first `go` on the Windows box was on `a38d549`; the first two were on `e5d8e73` and stopped at winget (live-s2 evidence/41, /42). IAP 4047 came 7 s and 19 s after create returned (live.md: 10:19:34Z to 10:19:41Z, 10:30:18Z to 10:30:37Z), not a minute. - behaviour:F7. "What did not run" now lists a Linux GPU install through comfy-cli, --torch, a --commit install, --no-manager, and nodes and models surviving a build change. - code:F2 / behaviour:F2. troubleshooting.md and the entry say the wrong-build fix names the whole build, Manager included; code:P2's refusals name options rather than commands. - behaviour:P1. README Status no longer says everything up to 1.2.0 ran on Windows without the two Windows paths 1.4.0 found broken in 1.2.0 to 1.3.1 (winget and Start-Process are in provision.py at v1.2.0, v1.3.0, `215052f`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * Docs examples put their notes above the command, not on the end of it Audit parking item behaviour:P4. Fifty-one command lines in the README and docs ended in a `# note`, the new machines.md switch example among them. Pasted into an interactive zsh without `interactivecomments`, which is how zsh starts, `#` and every word after it reach the command as arguments, and comfy-qat refuses them: the class 1.3.1 fixed for what the tool prints. Each note now sits on its own line above its command, and test_docs holds every ```sh block to it (red on the old docs with 51 lines; its own guard tells a note from a quoted `#`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A PR asked for again after new commits is fetched again, never handed over on its old head Audit finding code:F3. `go --pr 17497` on a box whose record named PR 17497 opened the browser on the commit the box checked out days ago: `same_choice` ignored the head GitHub reported and `contradictions` accepted the PR's branch at any commit. - `contradictions` names a PR branch at a commit that is not the head: "PR 17497 at 0000000000, not its head 9350a08d80". - `ensure_installed` and `_serve` read the checkout against the build asked for (`checkout_contradictions`) and change it when it is stale, so `comfy install --restore --pr` fetches the new head; the frontend-only shortcut is skipped then. - The box is confirmed against the build asked for, not its old record. - The host list keeps the choices only (`Build.choices`): a PR head written the day `create` ran would otherwise refuse the first `go` for being current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A release is held to its tag's commit, and a frontend PR is said to be unconfirmed Audit finding behaviour:F3. For `--version v0.43.0` "ready" compared only the version /system_stats reports, so a checkout on master's tip or on a PR that still says 0.43.0 was handed over as the release, under a stamp line that said `(PR 17497, 9350a08d80)`. - `check` keeps the commit the tag points at (`Build.tag_commit`), following an annotated tag to its commit; the box's record keeps it, the host list does not. - `contradictions` names a commit that is not the tag's, or a PR branch, for a release: "commit abcdef0123, not v0.43.0's commit 426f5eea52", "PR 17497, not v0.43.0". Through `checkout_contradictions` the checkout is put back before launch, as for a PR's old head. - A frontend PR, which nothing a running ComfyUI answers can confirm, is said to be unconfirmed rather than passing in silence. - The fake box checks a release out at its tag's commit, through the tag object for an annotated tag, as git does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A backend chosen with no frontend gets the frontend it pins, and --frontend-version pinned puts it back Audit finding behaviour:F8, with the lead's rule. S2-2 says "Giving none keeps the frontend the backend pins", and `Build.over` kept a recorded frontend release through a backend change: live, `switch --pr 17497` served 1.58.4 where the PR pins 1.57.6, and no option could put the pinned one back. - A command that chooses a backend (--version, --pr or --commit) and names no frontend gets the frontend that backend pins: no --front-end-version on the launch, and the record says frontend=default. - A command that chooses no backend keeps the frontend the box has; a bare `go` keeps the whole recorded build, as before. - `--frontend-version pinned` returns a box to the pinned frontend without changing the backend. It asks GitHub nothing, is handed on by --new-window, and is named in --help and in the refusal for a word that is not a tag. - --help, the CHANGELOG 1.4.0 entry and machines.md's "Choosing the build" table say this rule; machines.md says the live second stamp predates it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A create handed back by a refusal keeps the build Audit finding behaviour:F1. The same create offered after a stock-out, and after --zone with --region, was rebuilt from where the box goes, the name, --reserve and --disk, and dropped the build: pasted back, the box came up on the tip of ComfyUI's default branch with Manager on. Live, S2-13's retry lost `--version v0.43.0` (evidence/56). - `_same_box` carries the blueprint's build (`Build.flags`, --no-manager when Manager is off). - `besides_where`, `_asked_for` and `plan` take the build options as typed, so every refusal before the build is checked — --zone with --region, a wrong-case region, a disk too small or too large, no such OS, an undrivable card, a box with no GPU and --reserve, a missing --os or --gpu — hands back the same build. - The class: `_refuse_the_leftover` keeps the build and the disk, and `move`'s create-it-again for a reserved box carries the host list's `build`. - The O23 "same box" tests run every case again with --version, --frontend-version, --no-manager and --torch, and paste each command back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * A fresh install is held to the build chosen, not a record left from before code:F3's third step, on the install from nothing too. The box was confirmed against whatever record it carried when one was there, so an install script that exited 0 before writing its own left an earlier record in charge: asked for v0.43.0, a box that came up on 0.42.1 under a leftover v0.42.1 record was reported ready. The fake box gains `records=False` for that script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.4.0 entry says the four audit fixes ran only against the fakes Added to "What did not run", and nothing else in that paragraph changed: the PR fetched again after new commits, the release held to its tag's commit, the frontend a new backend pins, and the build kept by every create a refusal hands back were all made after the live run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The docs say a frontend PR is not fetched again after new pushes The class of code:F3, on the frontend: comfy-cli 1.24.0 keeps each frontend PR's build for seven days under its number, branch and author (pr_cache.py, `get_frontend_cache_path`, `is_cache_valid`), never its head, and serves it again after new pushes. Asking again for a ComfyUI PR now fetches its new commits; the docs no longer leave a reader to assume the same of a frontend PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The moved-head and tag tests hold the line that says so, and the frontend shortcut The serving box's own check in `_serve` was held by nothing: without it the record-contradiction path still changed the build, under "came up on a different build", which is not what happened. The tests now hold the line that is printed, once, and a PR asked for again with a frontend release, which without the `checkout_stale` guard is only relaunched on its old head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * "Only what is named changes" says the frontend follows a new backend The CHANGELOG's build-change bullet still said only what is named changes, which the frontend rule makes untrue for a new backend; it now says so. The machines.md paragraph the fixes grew is rewrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The 1.4.0 entry gives the IAP retries' real timing, and two more paths that never ran live On Linux the retry came about 40 s after the refused go, and on Windows it ran on code that already waited 4047 out (live-s2/live.md l.102, l.125, l.404, l.427). "What did not run" now also names a build change on a Windows box and the wrong-build refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz * The wrong-build entry gives the real causes of a PR off its head A failed fetch does not leave a checkout on the PR's branch behind its head: comfy-cli 1.24.0's checkout_pr returns False on a failed `git fetch` (git_utils.py), install exits 1, and the change is reported as not finished. What does leave it is a push between the head comfy-qat read and the fetch on the box, or a checkout moved by hand. The entry now says so, and what to do after a push, including that 1.4.0 keeps the older head in the box's record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbT49jkR4MQGRJLjx16xaz --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>		2026-11-04 01:28:14 +02:00
.github/workflows	ci: update checkout to v7 (#33 )	2026-09-30 20:28:09 +01:00
comfy_qa	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00
docs	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00
tests	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00
.gitignore	fix: earn the claim about what is billing, in both directions	2026-09-04 16:25:33 +01:00
CHANGELOG.md	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00
pyproject.toml	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00
README.md	1.4.0: choose the build (backend, frontend, Manager on by default, PyTorch), installed by comfy-cli, and the Windows NVIDIA driver installed for you (#42 )	2026-11-04 01:28:14 +02:00

README.md

comfy-qa-tools

QA tooling for testing Comfy.

Most testing time is not spent testing. It goes on standing a machine up, getting it into a state where a result means something, and working out which box or which build you were actually looking at. This tool is that layer. Less setup, less environment wrangling, less doubt about what you are pointed at, more time testing.

Features land one at a time and are documented here when they ship — so what is on this page is what the binary does today.

Not the same tool as `comfy-qa`

Comfy-Org/Comfy-QA, maintained by snomiao, does AI-driven E2E test runs. The similar names are a coincidence — two different projects, neither replacing the other. This one's binary is comfy-qat, so they never collide on PATH.

Status

1.4.0. Every command is implemented and tested, and everything up to 1.2.0 has run against real GCE hardware on Linux and Windows, except two Windows paths the 1.4.0 live run found broken in 1.2.0 to 1.3.1: installing ComfyUI on a box from nothing (it called winget, which Windows Server 2022 does not have) and the detached launch (a process started with Start-Process did not outlive the SSH call that started it). Both are fixed in 1.4.0; see the changelog. 1.4.0 lets you choose the build — a ComfyUI release, PR or commit, a frontend, Manager (on by default) and a torch pin — installs it with comfy-cli on the box, shows the build that is running in stamp, and has a Windows GPU box install its own NVIDIA driver. The build choice ran on a real Linux box and a real Windows box, both with no GPU, on 2026-11-03; a Windows GPU box installing its own driver has not run on a real box yet — see the changelog for what did and did not.

What 1.3.0 adds — reserved boxes, the limit on them, boxes with no GPU, and the cost columns in list — was run on real Google Compute Engine boxes on 2026-10-07: a reserved L4 box, a box with no GPU, the limit, list and delete. A reserved T4 box could not be run, for lack of stock, and nothing that needs more than one GPU of quota has been run for real.

The verbs are at the top level: comfy-qat go linux. The host ... and auth ... spellings were a deprecation window rather than a second permanent way to type everything, and it closed at 1.1.0 — typing one now exits 2. An old run sheet that says host create or auth status wants the noun taken off.

env is carried over from v0, hidden, and awaiting its own release.

Install

Needs Python 3.11 or newer.

git clone https://github.com/Comfy-Org/comfy-qa-tools.git
cd comfy-qa-tools

Into an existing virtualenv — the fewest moving parts, and it works with what you already have:

/path/to/venv/bin/pip install -e .

-e runs it from the checkout, so git pull updates it with no reinstall. The binary lands next to that venv's other scripts, at /path/to/venv/bin/comfy-qat. If that directory is not on your PATH — venv bin directories usually are not — either call it by full path or add an alias:

alias comfy-qat='/path/to/venv/bin/comfy-qat'

Or standalone with uv, which keeps it out of every virtualenv:

uv tool install .

Two things to check if comfy-qat is then "not found": uv itself has to be on your PATH (it is often installed inside a virtualenv rather than system-wide), and uv puts binaries in ~/.local/bin, which is not on PATH by default on macOS. uv tool update-shell fixes the second.

Upgrading from an earlier install

Earlier versions installed as comfy-qa-cli or comfy-qa. Remove them — the second in particular installs a comfy-qa binary, which is a different project's command:

pip uninstall -y comfy-qa-cli comfy-qa

Then update any shell alias to point at comfy-qat.

First run

One command.

comfy-qat setup

It signs you in to Google Cloud, picks your project, checks billing, sorts out GPU quota and writes your host list — saying what it is doing, and asking only where the decision is genuinely yours. If something needs you (no billing account, no project), it stops and prints the link. Run it again afterwards; it skips what is already done.

It also asks Google for the GPU quota your project is missing, so a fresh install ends with every card this tool can drive either granted or with a request in flight. The exact list is printed before anything is sent — a quota request cannot be withdrawn and may be read by a person at Google — and --no-quota-request skips it entirely.

It works without prompts too: --project, --region, --non-interactive.

# see your machines
comfy-qat list
# re-check readiness at any time
comfy-qat status

Everyday use

# start it, tunnel in, leave ComfyUI running on the box
comfy-qat go windows
# what that ComfyUI is saying, whenever you want it
comfy-qat logs windows
# the line that says what produced your result, build included
comfy-qat stamp windows
# close the tunnel, stop the box, stop paying
comfy-qat down windows
# stop every cloud box you have declared
comfy-qat down --all

windows there is not a special name — it is the machine described rather than named. Anywhere a host name goes you can say what you want instead: an operating system (windows, linux, ubuntu, debian, macos), a card (l4, a100), or both (windows/l4). Whatever it resolves to is printed, one host per line:

windows -> comfy-win (Windows Server 2022, L4)

If two machines fit, it refuses and lists them with their OS and card rather than picking one. Names always win over a description.

Changing machine is one command, which stops the box you were on:

# start the Linux box, then stop the Windows one
comfy-qat switch linux

Making the box is one command too, and the card is the only real decision:

comfy-qat create --os linux --gpu t4

The machine type follows from the card — an L4 is a G2 with the GPU built in, a T4 is an N1 with one attached — and the zone is chosen rather than typed: regions this project holds quota in, zones inside them that offer the card and the machine type, ranked by latency measured from your machine, and tried in order until one has capacity. Quota is checked before anything exists, both the card's own grant and GPUS_ALL_REGIONS, the project-wide ceiling that is the one that usually bites. --dry-run prints the plan, the quota it read and the zone order it would try, and creates nothing. The finished box is added to your host list on a free port, so go works on it straight away.

The NVIDIA driver is not in either base image, and a box without one runs ComfyUI on its CPU while looking perfectly healthy. Both kinds of box install it themselves on first boot, each with Google's own documented method: Linux from a startup script, Windows from Google's driver script set as the box's sysprep-specialize-script-url. Nothing is left for you to run, and go waits for the driver on both.

The build is a choice. create, go and switch take a ComfyUI release, PR or commit (--version v0.43.0, --pr 17497, --commit 426f5ee), a frontend release or PR (--frontend-version v1.58.4, --frontend-pr 5678), and a torch pin (--torch 2.12.1+cu130); ComfyUI-Manager is on unless you say --no-manager. comfy-cli does the installing, on the box. A build that does not exist is refused before anything is created, a new build on an existing box keeps its custom nodes and models, and go reports a box ready only once the ComfyUI running on it, and git in its checkout, say it is the build you asked for. With none of them, it is the tip of ComfyUI's default branch, as before. Choosing the build has the whole of it.

go is the one command worth memorising. It starts the instance, installs ComfyUI if the box has none — the build you chose, with comfy-cli, and a torch built for that box's CUDA — launches it on the box, and forwards a local port to it as soon as it is listening. The forward is an ssh -L over Identity-Aware Proxy: it reaches the box's own loopback, so nothing is exposed on any interface and no firewall rule is involved.

On a box you have just created, go waits for it to finish starting. Google reports a machine as RUNNING the moment it is powered on, which is some minutes before it accepts SSH — and on Linux the driver install reboots it once or twice on top of that. A box that is genuinely ready is not delayed by this; a box that is still coming up says so while it waits, rather than failing on the tunnel.

ComfyUI stays running there and you get your prompt back. It used to be launched in the foreground with its log streamed onto your terminal, which meant one machine per terminal and a Ctrl-C that killed ComfyUI. Now logs <host> reads that log whenever you want it, --follow brings the old streaming back for when you are debugging a launch, --new-window hands the whole thing to a new macOS Terminal window, and two boxes can be up at once from one prompt. What has not changed: go still does not return until ComfyUI has really answered on the tunnel, because a box that booted and serves nothing bills exactly like one that works.

down is what stops the billing — and down --all stops every declared cloud box, because the question at the end of a session is never "is comfy-win stopped", it is "am I still paying for anything".

docs/machines.md covers the whole loop, including what to do when a zone has no GPUs left.

Commands

Every target is a declared host, local or cloud — naming them all is the point, so local stops being an invisible default and picking the wrong machine becomes something you do on purpose.

command	what it does
`comfy-qat setup`	get this machine ready, in one command
`comfy-qat status`	signed in? which project? billing and GPU quota?
`comfy-qat login`	print the sign-in commands
`comfy-qat quota list`	what cards you can start, and where — `--by-region`
`comfy-qat quota request`	ask Google for cards — `--gpu l4,a100 --region us-central1`
`comfy-qat init`	write a starter host list you can edit
`comfy-qat list`	every declared machine, what is up and what is reserved — `--live` asks Google, and adds age and disk
`comfy-qat discover`	find cloud boxes and add them — `--prune` drops the entries Google confirms are gone
`comfy-qat create`	make a box; the zone is chosen, not typed — `--reserve` holds its capacity, `--gpu none` has no GPU
`comfy-qat delete`	remove a box and its disk, permanently — and release its reservation, if it has one
`comfy-qat go`	start it, make sure ComfyUI is on it, hand the prompt back
`comfy-qat up`	start a machine, and succeed only if ComfyUI is already serving
`comfy-qat open`	tunnel to a machine already running
`comfy-qat disconnect`	close the tunnel, leave the machine running
`comfy-qat down`	close the tunnel and stop the machine, so it stops costing — a reserved box goes on billing, and `down` says so
`comfy-qat ssh`	a shell on a box, through the tunnel
`comfy-qat rdp`	reset the Windows password and forward RDP
`comfy-qat logs`	follow the ComfyUI log on a box — `--tail N` reads the end and stops
`comfy-qat stamp`	ask a machine what it is, and print the line you paste
`comfy-qat switch`	start the one you want, stop the one you were on
`comfy-qat move`	move a box to a zone with capacity, keeping its ComfyUI — not a reserved one
`comfy-qat --version`	what you are running, with the commit

docs/commands.md has the flags and which to reach for. comfy-qat <command> --help says the same at the prompt.

The host list

Lives at ~/.config/comfy-qa-tools/hosts.toml. comfy-qat setup writes it and fills in your cloud boxes automatically — Google already knows each one's zone, card and operating system, so none of it needs typing. comfy-qat discover does the same on demand, and never changes an entry you already have.

discover --prune reconciles the other way, and it is the only thing here that removes one. Boxes disappear without this tool — deleted from the console, by a colleague, by raw gcloud — and a stale entry is not untidiness, because it holds a name and a port that nothing else can use. Removal takes two answers from Google that agree: the name is nowhere on the project, and then a direct describe of that one machine comes back not-found. Anything else is reported and kept, including the case that matters most — a box that turns up in a zone other than the one your entry gives is a wrong entry, not a missing machine, and deleting it would leave a running GPU with nothing naming it.

[hosts.local]
kind = "local"
port = 8188

[hosts.comfy-linux]
kind         = "gce"
os           = "Ubuntu 22.04"
gpu          = "L4"
gce_instance = "comfy-linux"
gce_zone     = "us-central1-a"
gce_project  = "your-project-id"
port         = 8190

Seven rules are enforced offline, before anything else runs. The two you meet first:

No cloud host may use 8188. That is the local ComfyUI's port; a tunnel on it would silently point you at the wrong machine, so it is refused, not warned about.
No two hosts may share a port. If two do, you cannot tell which one you reached.

The other five — one box per cloud instance, no two names differing only in case, local reserved for this machine, no gce_ fields on a local host, and a name that is actually typeable — are in docs/hosts.md with the reason for each.

Unknown fields are rejected rather than ignored, and the message names the typo and what it was probably meant to be:

host 'comfy-linux': unknown field(s) 'gce_zoen' (did you mean 'gce_zone'?). Known fields: build, gce_instance, gce_project, gce_reservation, gce_zone, gpu, kind, os, port.

Switching OS means switching host: one box per OS, and nothing is ever reimaged. os and gpu are what windows, l4 and windows/l4 match on, so with one box per OS you never have to remember what you called it. Every field is documented in docs/hosts.md.

What it writes

Everything lives under ~/.config/comfy-qa-tools/:

hosts.toml — the host list, and the only file you would ever edit.
hosts.toml.bak, and backups/ — anything that rewrites the host list copies it first, reads the copy back, and refuses the rewrite if it could not be made. .bak is the file as it was before the command you last ran; the copy it supersedes is archived under a timestamp in backups/ rather than overwritten, and that directory is capped.
zone-latency.json — how long a TCP connect to each Google Cloud region took from this machine, measured by create and reused for a week. Delete it to re-measure; nothing else reads it.
tunnels/<host>.pid, tunnels/<host>.json and tunnels/<host>.log — written by open, up and go. A tunnel outlives the command that started it, so what it is gets recorded rather than assumed: the pid, the moment that process started, and which instance, zone and port it goes to. Pids are recycled, so the number alone is not an identity and a record that no longer fits is treated as stale rather than trusted. The log is what gcloud said while opening it. down and disconnect remove the .pid and the .json — the two that claim a tunnel is there — and keep the .log, so you can still read why one died after you have closed it. Each open starts that log fresh, so it holds the most recent tunnel for that host and nothing older.

Signing in is gcloud's job, so credentials live in gcloud's own store (~/.config/gcloud/) and are refreshed by it. This tool never sees, stores or prints a credential.

It does not touch your ComfyUI installs, your PATH, your shell config, or any service on this machine. It will not overwrite a host list you already have, and running setup again is safe — it skips whatever is already done.

On a cloud box, go does install software: ComfyUI into C:\ComfyUI (Windows) or /opt/comfyui (Linux), with a Python 3.12 environment beside it — and on Windows, Git, Python 3.12 and the Visual C++ runtime themselves, from their publishers' own installers, each checked before it runs (by its pinned SHA-256, or the runtime by Microsoft's signature). That only ever happens on a declared gce host, never locally.

Docs

getting started · every command · the everyday loop · the host list · troubleshooting · cost · when your session expires · test criteria · spot instances · tests that cannot fail

Design

Each feature is a sibling sub-app, added in one line. comfy-qat <feature> <action>; a new feature never touches an existing one.

comfy-cli has no plugin mechanism — no entry-points table, all sub-apps registered by static add_typer calls — so this ships as its own binary. register() in comfy_qa/cli.py is the object a plugin entry point would take if comfy-cli ever grows one; the reservation is already in pyproject.toml. Nothing here depends on that happening.

Tests

pytest tests/

The docs are tested too: every error the tool can print has to appear in troubleshooting.md, and every command in the CLI has to appear in this README. Drift between the code and the page is a failing test, not something to notice six weeks later.