mirror of
https://github.com/Comfy-Org/comfy-python-sdk.git
synced 2026-11-04 23:37:47 +00:00
[PR #145] [MERGED] fix: return a model run's native bytes instead of raising on a non-JSON 200 #145
Labels
No labels
audit:ai-readiness
audit:community
audit:github
audit:health-scoring
audit:policy
audit:release
audit:supply-chain
effort:medium
effort:small
effort:small
effort:trivial
priority:nice-to-have
priority:nice-to-have
priority:recommended
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/comfy-python-sdk#145
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
📋 Pull Request Information
Original PR: https://github.com/Comfy-Org/comfy-python-sdk/pull/145
Author: @mattmillerai
Created: 9/11/2026
Status: ✅ Merged
Merged: 10/3/2026
Merged by: @mattmillerai
Base:
main← Head:matt/be-13346-binary-model-run-result📝 Commits (5)
f566a9efix: return a model run's native bytes instead of raising on a non-JSON 200a26ed9aMerge branch 'main' into matt/be-13346-binary-model-run-result3ab6d45fix(models): harden the run-result branch against the review findings1563c21merge main into matt/be-13346-binary-model-run-result, resolve conflicts943f8acfix(models): return BinaryResult from the queued result route too📊 Changes
13 files changed (+1223 additions, -116 deletions)
View changed files
📝
CHANGELOG.md(+65 -1)📝
README.md(+63 -3)📝
scripts/check_drift.py(+52 -6)📝
src/comfy_low/__init__.py(+2 -1)📝
src/comfy_low/transport.py(+260 -32)📝
src/comfy_sdk/__init__.py(+2 -0)📝
src/comfy_sdk/model_requests.py(+23 -19)📝
src/comfy_sdk/models.py(+54 -15)📝
tests/conftest.py(+101 -11)📝
tests/test_models_queue.py(+44 -1)📝
tests/test_models_run.py(+2 -2)➕
tests/test_models_run_binary.py(+491 -0)📝
tests/test_router_spec_contract.py(+64 -25)📄 Description
ELI-5
Some models on Comfy Router hand back a file, not a JSON document — the two ElevenLabs models answer a generation as raw
audio/mpegbytes. The SDK assumed every successful response was JSON, tried to decode the MP3 as text, and blew up withComfyError: Could not decode the 200 response body as JSON. The generation had already run and already been billed; the SDK threw the audio away. Nowmodels.run()looks at the response'sContent-Typefirst: JSON still comes back as adictexactly as before, and anything else comes back as aBinaryResultyou can write straight to a file.What changed
comfy_low.transport._Prepared.parse_run_resultis a new sibling ofparse_or_raise, used bypost_model_runand nothing else. On a success status it branches on the responseContent-Type:application/json, or a+jsonsuffix typedict— unchanged, including{}for an empty body andinvalid_responsewhen the body will not parseaudio/mpeg,audio/L16; rate=16000, ...)BinaryResult(content, content_type, request_id)Content-Typeat all{}if the body is empty,dictif it parses as a JSON object, otherwiseBinaryResultwithcontent_type=""BinaryResultis a frozen dataclass exported from bothcomfy_sdkandcomfy_low, exposing exactly the three fields the design called for. The bytes are the partner's file verbatim — not base64-encoded, not wrapped in a dict, not decoded or transcoded.content_typekeeps the header's parameters, because for some partner media types the parameters are part of what the bytes are: ElevenLabs'pcm_*output formats arrive asaudio/L16; rate=16000and the sample rate is not decoration. It is bounded to 128 characters and stripped of unprintable characters first — the same filter every other server-supplied string this SDK surfaces goes through — which no real media type is affected by.Models.run/AsyncModels.runare now annotateddict[str, Any] | BinaryResult, andRouterRunResult.output— therun_detailedshape that landed onmainwhile this branch was open — carries the same union.post_model_runand_runanswertuple[dict[str, Any] | BinaryResult, Mapping[str, str]], which is the merge of the two intents:mainmade them return(body, headers)sorun_detailedcan disclose the fallback provider and dropped params, and this branch made the body itself a union. The queued surfacemainadded keepsparse_or_raise, since JSON is the only success media type its routes declare. Nothing moves out oftranslating(...), so a failure on the binary path still carries.idempotency_keyand anIdempotent-Replayedbinary 200 comes back the way a first run does.scripts/check_drift.pyandtests/test_router_spec_contract.pyalso gained a comparison of the media types the run route's 200 declares against the two the SDK branches on — so a future spec sync that drops the*/*branch, or adds a third one, fails CI rather than leaving the branch silently dead or silently incomplete.Chesterton's Fence — the guard this changes, and why the history says it is right
parse_or_raise's "a proxy interstitial served as 200" reading is untouched and still serves every other operation. It also still fires on the run route, for the case it was actually written for: aContent-Typethat claims JSON over a body that does not parse. That branch is where the reading belongs, because there the response promised a document and did not deliver one.What the run route gives up is that reading for a non-JSON media type. The commit that introduced the guard did so to make a lost result translatable and stamped — "on
models.runthis is a generation that ran and was billed with the result lost, which is exactly the failure theIdempotency-Keyhas to ride out on." On a binary 200 the result is not lost: it is sitting in the body. So honouring the declared media type serves the original intent rather than working against it, and the SDK cannot distinguish an interstitial from a partner's own text output anyway. Atext/html200 now reaches the caller as bytes they can inspect, which is strictly more information than an exception that discards them.Self-review notes
if media: ... return self._binary_result(resp)— it is the one place an existingraisebecame areturn. It is bounded topost_model_run:parse_or_raiseis byte-for-byte the same behaviour for all eleven other operations, whose routes declareapplication/jsonas their only success media type. The branch is on the declared type rather than on whether the bytes parse, so a JSON model cannot fall into it (test_a_json_content_type_with_a_charset_is_still_the_dict_branch,test_a_json_suffix_media_type_is_the_dict_branch), andapplication/jsonlinesis explicitly not JSON — astartswithtest would have decoded it as a document.post_model_runhas exactly two consumers, both incomfy_sdk/models.py, both of which return the transport's value directly. Nothing else insrc/reads a run result, so there is no caller left indexing adictthat might now be aBinaryResult.__repr__is hand-written on purpose. A frozen dataclass's default repr would print every byte of a multi-megabyte audio file into a traceback, a REPL echo or a CI log. Asserted (test_binary_result_repr_does_not_dump_the_body).raisethe run route keeps is the pre-existing JSON-that-will-not-parse one.models.runwill now be asked to narrow. That is deliberate — it is the point of the annotation — and the SDK is pre-1.0, so it is noted in the changelog under Added rather than gated behind a flag. There is no runtime flag mechanism in this SDK, and the current behaviour is an exception rather than a contract anyone can depend on.How this was verified
spec/router-openapi.yamlis byte-identical to the canonical Router contract it is vendored from (diffreturned no output), and the*/*format: binarybranch onrunRouterModel's 200 is already present there — it landed in the last sync. So there is no spec diff in this PR; the file is already current, and the new drift check is what keeps it honest from here.elevenlabs/eleven_v3declares its 200 ascontent: {"*/*": {type: string, format: binary}}with the description "Raw audio bytes. The Content-Type and encoding follow the requested output_format and are forwarded from ElevenLabs."runRouterModel200 content maps inspected: 2 declare aformat: binary200 (elevenlabs/eleven_v3,elevenlabs/eleven_sfx_v2) and the remaining 205 areapplication/json-only. Those 205 are exactly the models whose result shape this PR must not change, and the JSON branch is unchanged for all of them.mainand applying the review fixes:pytest— 1034 passed, 9 skipped (56 of the passing tests are the binary-200 file).ruff check .clean,ruff format --check .clean (58 files),mypy srcclean (21 files),scripts/check_drift.pyall three checks OK,scripts/check_public_repo_hygiene.pyOK. All 12 PR checks green on3ab6d45, including the 3.10–3.13 matrix.Residual
Not verified against the live service. Every test here drives the repo's stdlib stub server. The reported repro —
client.models.run("elevenlabs/eleven_v3", ...)against production Router — was not run, because a live run is a billed generation against a paid partner API and this environment holds no credential for it. The stub reproduces the reported failure shape exactly (200,audio/mpeg, an ID3 header followed by a0xff 0xfbMP3 frame sync — the same0xffthe originalUnicodeDecodeErrornamed), and the server contract was verified statically as described above, but nobody has yet confirmed end to end that a realeleven_v3call returns audio through this SDK. A human with a Router credential should make one liveeleven_v3call before this is treated as closed.elevenlabs/eleven_sfx_v2is untouched and still expected to fail. It is one of the two binary models and it is named in the report, but its failure is a422on the Router route itself — a server-side binding bug tracked separately, not something this SDK change can fix. This PR makes the SDK ready for it; the model will still not run until the server side lands.eleven_v3is the one to verify against in the meantime.Buffered, not streamed. The whole binary body is read into memory (
resp.content) beforeBinaryResultis constructed. That matches whatraw_requestalready did for every other operation and was explicitly out of scope, but it means a large binary result is fully resident in the client process. Raised by three of the six panel reviewers and deferred to a tracked follow-up rather than dropped: closing it means adding a new public shape (a streaming accessor over the existingopen()hatch) and settling how it interacts with a retry loop that cannot rewind a partly-consumed stream. The follow-up separates the shippable half (a configurable body-size cap) from the half that needs a design call.No decoding or transcoding. The SDK hands back bytes and a content type and nothing more. Deliberate, and stated here so it is not mistaken for an oversight.
Sibling SDKs are unfixed. The same JSON-only assumption is present in the TypeScript SDK (
src/sdk/models.ts) and the Go SDK (comfylow/transport.go'sPostModelRun, which returnsmap[string]any). Both are separate repositories and separate tickets; nothing here changes them, so the two ElevenLabs models remain unusable from those two SDKs.Artifacts named in the report that could not be read. The originating trace lives in an internal observability tool that is not reachable from this environment, and the sibling/related issues referenced in the report were named to me by identifier only — their bodies were never fetched and are not reachable from here. None of them is quoted or linked in this PR, since this repository is public. The conclusions above rest on the two artifacts that were read directly: the upstream spec file and the per-model schema.
Provenance
3ab6d45(merge ofmain+ the review fixes) —pytest: 1034 passed, 9 skipped;ruff check .: clean;ruff format --check .: 58 files already formatted;mypy src: no issues in 21 source files;scripts/check_drift.py: 3/3 OK;scripts/check_public_repo_hygiene.py: OK; all 12 PR checks green including the 3.10–3.13 matrix, codegen drift and build-check. From the opening run, unchanged by the merge: upstream spec diff byte-identical; 207-schema corpus sweep 2 binary / 205 JSON-onlymain, which had meanwhile changedpost_model_run/_runto return(body, headers)forrun_detailed— merged ina26ed9a, resolving that against this branch'sdict | BinaryResultby carrying both (no test of either side was weakened); the panel'sX-Comfy-Request-Idgate and the empty-binary-body raise were declined rather than applied, with the reasoning on the threads and now in the code, because both would discard a billed generation when an intermediary strips a header or a partner serves an empty file; the streaming/size-cap finding was deferred to a follow-up; no live call was made against production Router (billed generation, no credential in this environment) — see## Residual;spec/router-openapi.yamlis not modified because it was verified already byte-identical to upstream, so the "re-sync" acceptance item is satisfied by verification rather than by a diffSummary by CodeRabbit
New Features
BinaryResulttype for accessing binary response data.Bug Fixes
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.