[PR #9] [MERGED] feat(serialization): externalize oversize image batch envelopes via per-frame presigned URIs #9

New issue

Closed

opened 2026-10-26 22:28:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:28:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-remote-nodes/pull/9
Author: @Kosinkadink
Created: 5/15/2026
Status: ✅ Merged
Merged: 5/15/2026
Merged by: @Kosinkadink

Base: main ← Head: feat/batch-envelope-externalization

📝 Commits (1)

6433a46 feat(serialization): externalize oversize image batch envelopes via per-frame presigned URIs

📊 Changes

3 files changed (+131 additions, -3 deletions)

View changed files

📝 client.py (+11 -0)
📝 protocol.py (+14 -0)
📝 serialization.py (+106 -3)

📄 Description

Summary

The pre-existing serialization.maybe_externalize only inspected the singleton data field, so a multi-frame frames=[…] IMAGE envelope whose summed base64 payload crossed the 8 MiB DEFAULT_MAX_INLINE_PAYLOAD_BYTES cap silently rode the wire as a 30+ MiB POST body — defeating the entire reason the cap exists. A 9× 2048² Flux2 reference batch hits ~200 MB base64 inline; that's the worst-case the new path is designed for.

This PR adds the client side of the lockstep fix: detect oversize multi-frame batches (sum of per-frame raw bytes), upload each frame as an individual presigned PUT in parallel via asyncio.gather, and rewrite the envelope to the new image:png_base64_batch_uri encoding with a parallel uris list. Server side closes the loop with a matching encoding-string dispatch in resolve_image_envelope_pngs.

Per-frame URIs (NOT a tar/zip blob) compose with the existing per-frame fetch helpers and let the server fetch in parallel.

Changes

`protocol.py`

Mirror new Capability.IMAGE_PNG_BASE64_BATCH_URI = "image:png_base64_batch_uri" token next to IMAGE_PNG_BASE64_BATCH. Comment block matches the server-side copy so the two stay byte-comparable.

`client.py`

Append Capability.IMAGE_PNG_BASE64_BATCH_URI to CLIENT_CAPABILITIES so the server knows this client can emit the externalised batch shape. Comment notes the opportunistic posture (any input that already accepts png_base64_batch transparently accepts the externalised variant — no descriptor opt-in needed).

`serialization.py`

maybe_externalize now branches on payload shape: if envelope["frames"] is a populated list, dispatch to the new _maybe_externalize_batch helper; otherwise fall through to the existing singleton-data path. Docstring expanded to describe both shapes and the encoding-flip-on-spill behaviour.
New _maybe_externalize_batch helper sums per-frame raw bytes (len(b64) * 3 // 4), short-circuits when the batch fits under the cap, and otherwise uploads every frame in parallel via asyncio.gather (preserves frame ordering — critical because Flux2 maps uris[k] → input_image_{k+1} 1:1) before rewriting the envelope to {type, encoding: png_base64_batch_uri, uris: [...], shape, dtype, color_space} with frames stripped.
import asyncio added at module top so asyncio.gather is available without per-call imports.

Verification

AST parse of protocol.py, serialization.py, client.py succeeds.
In-process round-trip test (with a fake comfy_api_nodes.util.client shim and a stubbed upload_asset):
- Under-cap batch envelope (1× 5 MiB raw, cap=8 MiB) → passes through unchanged (object identity preserved).
- Over-cap batch envelope (3× 5 MiB raw, cap=8 MiB) → triggers 3 PUTs via the stub, returns {type: image, encoding: png_base64_batch_uri, uris: ['…/asset/1','…/asset/2','…/asset/3'], shape: [3,1,1,3], dtype: uint8, color_space: srgb} with frames stripped.
- Singleton data envelope (10 MiB raw, cap=8 MiB) → still gets the legacy data → uri swap (existing path unaffected).
- Capability.IMAGE_PNG_BASE64_BATCH_URI in CLIENT_CAPABILITIES holds.

Lockstep

Paired server PR: https://github.com/Comfy-Org/comfy-rnp-server/pull/14 (feat/batch-envelope-externalization).

Server PR was opened first because the wire shape needs a server resolver to be meaningful — the orchestrator merges them in lockstep order.

Capability gating posture: IMAGE_PNG_BASE64_BATCH_URI is opportunistic — this client emits it whenever a batch envelope it produced overflows the inline cap, regardless of whether the descriptor's input_serialization lists it. Server accepts on any image input that already accepts png_base64_batch via the shared resolve_image_envelope_pngs dispatch. No new descriptor fields, no per-provider opt-in, no required_capabilities changes — matches the spec's "Default: clients advertise the cap, server accepts it on any image input that already accepts png_base64_batch. Don't add a new descriptor demand unless a provider needs it."

Output-side decoder: server providers emit png_base64_batch (or single-frame png_base64) on responses today; the URI shape is consumed on input only, so the client decoder (_DECODERS / _decode_image_batch_envelope) is unchanged. The existing decode_image_envelope already dispatches on the encoding string (not field presence), so adding a png_base64_batch_uri decode branch is a drop-in if a future provider starts emitting the URI shape on outputs. Logged as a known follow-up.

Do NOT merge until the paired server PR is merged — the orchestrator merges them in lockstep order.

Amp-Thread-ID: T-019e29b4-d606-7278-8584-52910661c981

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-remote-nodes/pull/9 **Author:** [@Kosinkadink](https://github.com/Kosinkadink) **Created:** 5/15/2026 **Status:** ✅ Merged **Merged:** 5/15/2026 **Merged by:** [@Kosinkadink](https://github.com/Kosinkadink) **Base:** `main` ← **Head:** `feat/batch-envelope-externalization` --- ### 📝 Commits (1) - [`6433a46`](https://github.com/Comfy-Org/comfy-remote-nodes/commit/6433a46c933dc58882d2312fae6d90c08432f423) feat(serialization): externalize oversize image batch envelopes via per-frame presigned URIs ### 📊 Changes **3 files changed** (+131 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `client.py` (+11 -0) 📝 `protocol.py` (+14 -0) 📝 `serialization.py` (+106 -3) </details> ### 📄 Description ## Summary The pre-existing `serialization.maybe_externalize` only inspected the singleton `data` field, so a multi-frame `frames=[…]` IMAGE envelope whose summed base64 payload crossed the 8 MiB `DEFAULT_MAX_INLINE_PAYLOAD_BYTES` cap silently rode the wire as a 30+ MiB POST body — defeating the entire reason the cap exists. A 9× 2048² Flux2 reference batch hits ~200 MB base64 inline; that's the worst-case the new path is designed for. This PR adds the **client side** of the lockstep fix: detect oversize multi-frame batches (sum of per-frame raw bytes), upload each frame as an individual presigned PUT in parallel via `asyncio.gather`, and rewrite the envelope to the new `image:png_base64_batch_uri` encoding with a parallel `uris` list. Server side closes the loop with a matching encoding-string dispatch in `resolve_image_envelope_pngs`. Per-frame URIs (NOT a tar/zip blob) compose with the existing per-frame fetch helpers and let the server fetch in parallel. ## Changes ### `protocol.py` - Mirror new `Capability.IMAGE_PNG_BASE64_BATCH_URI = "image:png_base64_batch_uri"` token next to `IMAGE_PNG_BASE64_BATCH`. Comment block matches the server-side copy so the two stay byte-comparable. ### `client.py` - Append `Capability.IMAGE_PNG_BASE64_BATCH_URI` to `CLIENT_CAPABILITIES` so the server knows this client can emit the externalised batch shape. Comment notes the opportunistic posture (any input that already accepts `png_base64_batch` transparently accepts the externalised variant — no descriptor opt-in needed). ### `serialization.py` - `maybe_externalize` now branches on payload shape: if `envelope["frames"]` is a populated list, dispatch to the new `_maybe_externalize_batch` helper; otherwise fall through to the existing singleton-`data` path. Docstring expanded to describe both shapes and the encoding-flip-on-spill behaviour. - New `_maybe_externalize_batch` helper sums per-frame raw bytes (`len(b64) * 3 // 4`), short-circuits when the batch fits under the cap, and otherwise uploads every frame in parallel via `asyncio.gather` (preserves frame ordering — critical because Flux2 maps `uris[k]` → `input_image_{k+1}` 1:1) before rewriting the envelope to `{type, encoding: png_base64_batch_uri, uris: [...], shape, dtype, color_space}` with `frames` stripped. - `import asyncio` added at module top so `asyncio.gather` is available without per-call imports. ## Verification - AST parse of `protocol.py`, `serialization.py`, `client.py` succeeds. - In-process round-trip test (with a fake `comfy_api_nodes.util.client` shim and a stubbed `upload_asset`): * Under-cap batch envelope (1× 5 MiB raw, cap=8 MiB) → passes through unchanged (object identity preserved). * Over-cap batch envelope (3× 5 MiB raw, cap=8 MiB) → triggers 3 PUTs via the stub, returns `{type: image, encoding: png_base64_batch_uri, uris: ['…/asset/1','…/asset/2','…/asset/3'], shape: [3,1,1,3], dtype: uint8, color_space: srgb}` with `frames` stripped. * Singleton `data` envelope (10 MiB raw, cap=8 MiB) → still gets the legacy `data → uri` swap (existing path unaffected). * `Capability.IMAGE_PNG_BASE64_BATCH_URI in CLIENT_CAPABILITIES` holds. ## Lockstep Paired server PR: **https://github.com/Comfy-Org/comfy-rnp-server/pull/14** (`feat/batch-envelope-externalization`). Server PR was opened first because the wire shape needs a server resolver to be meaningful — the orchestrator merges them in lockstep order. Capability gating posture: `IMAGE_PNG_BASE64_BATCH_URI` is **opportunistic** — this client emits it whenever a batch envelope it produced overflows the inline cap, regardless of whether the descriptor's `input_serialization` lists it. Server accepts on any image input that already accepts `png_base64_batch` via the shared `resolve_image_envelope_pngs` dispatch. No new descriptor fields, no per-provider opt-in, no `required_capabilities` changes — matches the spec's "Default: clients advertise the cap, server accepts it on any image input that already accepts `png_base64_batch`. Don't add a new descriptor demand unless a provider needs it." Output-side decoder: server providers emit `png_base64_batch` (or single-frame `png_base64`) on responses today; the URI shape is consumed on input only, so the client decoder (`_DECODERS` / `_decode_image_batch_envelope`) is unchanged. The existing `decode_image_envelope` already dispatches on the `encoding` string (not field presence), so adding a `png_base64_batch_uri` decode branch is a drop-in if a future provider starts emitting the URI shape on outputs. Logged as a known follow-up. Do NOT merge until the paired server PR is merged — the orchestrator merges them in lockstep order. Amp-Thread-ID: T-019e29b4-d606-7278-8584-52910661c981 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:28:52 +00:00