[GH-ISSUE #224] bug: Gemini streaming tool calls crash and lose thought signatures #137

New issue

Open

opened 2026-10-18 14:03:17 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-18 14:03:17 +00:00

Owner

Copy link

Originally created by @owahltinez on GitHub (Aug 17, 2026).
Original GitHub issue: https://github.com/open-webui/computer/issues/224

Summary

Gemini's OpenAI-compatible Chat Completions stream does not always use the exact chunk shape that stream_openai_completions() currently assumes.

In particular, Gemini can:

• omit tool_calls[].index;
• split function arguments across indexless deltas;
• finish a tool-call response with finish_reason: "stop" rather than "tool_calls";
• return extra_content.google.thought_signature, including as a standalone metadata-only delta.

The current implementation indexes tc["index"] directly and therefore raises KeyError: 'index'. If that lookup alone is made optional, the tool call can still be lost at finish_reason: "stop". If both are fixed, subsequent tool rounds can still fail because extra_content is discarded when the call is normalized, persisted, and reconstructed for history.

This breaks Gemini tool use through an OpenAI-compatible connection. A tool can be selected correctly, but Computer aborts before executing it or receives a Gemini 4xx validation error on the follow-up request.

Reproduction

Reproduced against current main at 9c54711.

A minimal representative stream is:

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"id":"function-call-1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Sydney\"}"},"extra_content":{"google":{"thought_signature":"REDACTED"}}}]},"finish_reason":null}]}
data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

At present, this reaches:

for tc in delta.get("tool_calls") or []:
    idx = tc["index"]  # KeyError: 'index'

Google's thought-signature rules also require the signature to be returned in the subsequent request exactly where it was received. For parallel function calls, it belongs to the first function call; all calls must precede all function responses. See Google's documentation: https://ai.google.dev/gemini-api/docs/generate-content/thought-signatures

Suggested fix

1. Merge tool-call deltas without assuming an index

Add a small accumulator in cptr/utils/ai.py. It preserves normal indexed OpenAI streams, matches indexless deltas by call ID, maps parallel indexless fragments by position, attaches a standalone Gemini signature to the first parallel call, and uses the last call only as the final single-call fallback:

def _merge_tool_call_delta(
    tool_calls: dict[int, dict],
    tool_call_delta: dict,
    last_index: int | None,
    *,
    position: int,
    batch_size: int,
) -> int:
    """Merge one OpenAI-compatible tool call delta and return its index."""
    index = tool_call_delta.get("index")
    call_id = tool_call_delta.get("id")

    if index is None and call_id:
        index = next(
            (
                existing_index
                for existing_index, current in tool_calls.items()
                if current.get("id") == call_id
            ),
            None,
        )
    if index is None and not call_id and batch_size > 1 and position in tool_calls:
        index = position

    extra_content = tool_call_delta.get("extra_content")
    google = extra_content.get("google") if isinstance(extra_content, dict) else None
    if (
        index is None
        and not tool_call_delta.get("function")
        and isinstance(google, dict)
        and google.get("thought_signature")
        and tool_calls
    ):
        index = next(iter(tool_calls))

    if index is None and last_index is not None:
        previous = tool_calls[last_index]
        if not call_id or not previous.get("id"):
            index = last_index
    if index is None:
        index = len(tool_calls)
        while index in tool_calls:
            index += 1

    current = tool_calls.setdefault(
        index,
        {"id": "", "name": "", "arguments_json": ""},
    )
    if call_id:
        current["id"] = call_id
    function = tool_call_delta.get("function") or {}
    if function.get("name"):
        current["name"] = function["name"]
    current["arguments_json"] += function.get("arguments", "")
    if "extra_content" in tool_call_delta:
        current["extra_content"] = copy.deepcopy(tool_call_delta["extra_content"])
    return index

Use it in the streaming loop and emit accumulated calls for either valid finish reason:

tool_calls: dict[int, dict] = {}
tool_calls_emitted = False
last_tool_call_index: int | None = None

# ... inside the chunk loop ...
tool_call_deltas = delta.get("tool_calls") or []
for position, tc in enumerate(tool_call_deltas):
    last_tool_call_index = _merge_tool_call_delta(
        tool_calls,
        tc,
        last_tool_call_index,
        position=position,
        batch_size=len(tool_call_deltas),
    )

finish_reason = choices[0].get("finish_reason") if choices else None
if (
    finish_reason in {"tool_calls", "stop"}
    and tool_calls
    and not tool_calls_emitted
):
    item = complete_reasoning_item()
    if item is not None:
        emitted = True
        yield {"type": "output", "item": item}
    for tc in tool_calls.values():
        event = {
            "type": "tool_call",
            "call_id": tc["id"],
            "name": tc["name"],
            "arguments": json.loads(tc["arguments_json"] or "{}"),
        }
        if "extra_content" in tc:
            event["extra_content"] = copy.deepcopy(tc["extra_content"])
        emitted = True
        yield event
    tool_calls_emitted = True

2. Preserve provider metadata through persistence and replay

When converting persisted function calls back to OpenAI messages in _output_items_to_messages():

if item.get("fc_id"):
    tc["fc_id"] = item["fc_id"]
if "extra_content" in item:
    tc["extra_content"] = item["extra_content"]

All four paths that create function-call items—automatic execution, queued approval, rejected/invalid calls, and ask_user—must retain the metadata. A helper avoids one path silently dropping it:

def _function_call_item(
    tool_call: dict,
    *,
    status: str,
    name: str | None = None,
    arguments: dict | None = None,
    **fields,
) -> dict:
    item = {
        "type": "function_call",
        "id": str(uuid.uuid4()),
        "call_id": tool_call["call_id"],
        "fc_id": tool_call.get("id", ""),
        "name": tool_call["name"] if name is None else name,
        "arguments": tool_call["arguments"] if arguments is None else arguments,
        "status": status,
        **fields,
    }
    if "extra_content" in tool_call:
        item["extra_content"] = tool_call["extra_content"]
    return item

Then replace the repeated inline function-call dictionaries with _function_call_item(...) in each path.

Tests

I reproduced this against the current main branch and have a focused standard-library test suite covering:

1. an indexless Gemini tool call ending with finish_reason: "stop";
2. ordinary indexed, fragmented OpenAI tool calls;
3. a single indexless fragmented call;
4. indexed parallel calls;
5. indexless fragmented parallel calls;
6. a standalone thought-signature delta attaching to the first parallel call;
7. persistence and replay of extra_content across a tool result.

All seven tests pass with the changes above, along with Ruff formatting and lint checks. I can provide the complete patch or a branch/PR if external contributions are accepted for this repository.

Originally created by @owahltinez on GitHub (Aug 17, 2026). Original GitHub issue: https://github.com/open-webui/computer/issues/224 ## Summary Gemini's OpenAI-compatible Chat Completions stream does not always use the exact chunk shape that `stream_openai_completions()` currently assumes. In particular, Gemini can: - omit `tool_calls[].index`; - split function arguments across indexless deltas; - finish a tool-call response with `finish_reason: "stop"` rather than `"tool_calls"`; - return `extra_content.google.thought_signature`, including as a standalone metadata-only delta. The current implementation indexes `tc["index"]` directly and therefore raises `KeyError: 'index'`. If that lookup alone is made optional, the tool call can still be lost at `finish_reason: "stop"`. If both are fixed, subsequent tool rounds can still fail because `extra_content` is discarded when the call is normalized, persisted, and reconstructed for history. This breaks Gemini tool use through an OpenAI-compatible connection. A tool can be selected correctly, but Computer aborts before executing it or receives a Gemini 4xx validation error on the follow-up request. ## Reproduction Reproduced against current `main` at `9c54711`. A minimal representative stream is: ```text data: {"choices":[{"index":0,"delta":{"tool_calls":[{"id":"function-call-1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Sydney\"}"},"extra_content":{"google":{"thought_signature":"REDACTED"}}}]},"finish_reason":null}]} data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: [DONE] ``` At present, this reaches: ```python for tc in delta.get("tool_calls") or []: idx = tc["index"] # KeyError: 'index' ``` Google's thought-signature rules also require the signature to be returned in the subsequent request exactly where it was received. For parallel function calls, it belongs to the first function call; all calls must precede all function responses. See Google's documentation: https://ai.google.dev/gemini-api/docs/generate-content/thought-signatures ## Suggested fix ### 1. Merge tool-call deltas without assuming an index Add a small accumulator in `cptr/utils/ai.py`. It preserves normal indexed OpenAI streams, matches indexless deltas by call ID, maps parallel indexless fragments by position, attaches a standalone Gemini signature to the first parallel call, and uses the last call only as the final single-call fallback: ```python def _merge_tool_call_delta( tool_calls: dict[int, dict], tool_call_delta: dict, last_index: int | None, *, position: int, batch_size: int, ) -> int: """Merge one OpenAI-compatible tool call delta and return its index.""" index = tool_call_delta.get("index") call_id = tool_call_delta.get("id") if index is None and call_id: index = next( ( existing_index for existing_index, current in tool_calls.items() if current.get("id") == call_id ), None, ) if index is None and not call_id and batch_size > 1 and position in tool_calls: index = position extra_content = tool_call_delta.get("extra_content") google = extra_content.get("google") if isinstance(extra_content, dict) else None if ( index is None and not tool_call_delta.get("function") and isinstance(google, dict) and google.get("thought_signature") and tool_calls ): index = next(iter(tool_calls)) if index is None and last_index is not None: previous = tool_calls[last_index] if not call_id or not previous.get("id"): index = last_index if index is None: index = len(tool_calls) while index in tool_calls: index += 1 current = tool_calls.setdefault( index, {"id": "", "name": "", "arguments_json": ""}, ) if call_id: current["id"] = call_id function = tool_call_delta.get("function") or {} if function.get("name"): current["name"] = function["name"] current["arguments_json"] += function.get("arguments", "") if "extra_content" in tool_call_delta: current["extra_content"] = copy.deepcopy(tool_call_delta["extra_content"]) return index ``` Use it in the streaming loop and emit accumulated calls for either valid finish reason: ```python tool_calls: dict[int, dict] = {} tool_calls_emitted = False last_tool_call_index: int | None = None # ... inside the chunk loop ... tool_call_deltas = delta.get("tool_calls") or [] for position, tc in enumerate(tool_call_deltas): last_tool_call_index = _merge_tool_call_delta( tool_calls, tc, last_tool_call_index, position=position, batch_size=len(tool_call_deltas), ) finish_reason = choices[0].get("finish_reason") if choices else None if ( finish_reason in {"tool_calls", "stop"} and tool_calls and not tool_calls_emitted ): item = complete_reasoning_item() if item is not None: emitted = True yield {"type": "output", "item": item} for tc in tool_calls.values(): event = { "type": "tool_call", "call_id": tc["id"], "name": tc["name"], "arguments": json.loads(tc["arguments_json"] or "{}"), } if "extra_content" in tc: event["extra_content"] = copy.deepcopy(tc["extra_content"]) emitted = True yield event tool_calls_emitted = True ``` ### 2. Preserve provider metadata through persistence and replay When converting persisted function calls back to OpenAI messages in `_output_items_to_messages()`: ```python if item.get("fc_id"): tc["fc_id"] = item["fc_id"] if "extra_content" in item: tc["extra_content"] = item["extra_content"] ``` All four paths that create function-call items—automatic execution, queued approval, rejected/invalid calls, and `ask_user`—must retain the metadata. A helper avoids one path silently dropping it: ```python def _function_call_item( tool_call: dict, *, status: str, name: str | None = None, arguments: dict | None = None, **fields, ) -> dict: item = { "type": "function_call", "id": str(uuid.uuid4()), "call_id": tool_call["call_id"], "fc_id": tool_call.get("id", ""), "name": tool_call["name"] if name is None else name, "arguments": tool_call["arguments"] if arguments is None else arguments, "status": status, **fields, } if "extra_content" in tool_call: item["extra_content"] = tool_call["extra_content"] return item ``` Then replace the repeated inline function-call dictionaries with `_function_call_item(...)` in each path. ## Tests I reproduced this against the current `main` branch and have a focused standard-library test suite covering: 1. an indexless Gemini tool call ending with `finish_reason: "stop"`; 2. ordinary indexed, fragmented OpenAI tool calls; 3. a single indexless fragmented call; 4. indexed parallel calls; 5. indexless fragmented parallel calls; 6. a standalone thought-signature delta attaching to the first parallel call; 7. persistence and replay of `extra_content` across a tool result. All seven tests pass with the changes above, along with Ruff formatting and lint checks. I can provide the complete patch or a branch/PR if external contributions are accepted for this repository.

bsakaguchi commented 2026-10-18 14:03:18 +00:00

Author

Owner

Copy link

@owui-terminator[bot] commented on GitHub (Aug 17, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

1. 🟣 #96 bug: OpenAI-compatible streaming usage can end responses early
   This reports a related OpenAI-compatible streaming bug where the implementation treats an intermediate event as final and ends the response too early. The new issue also centers on incorrect assumptions about streamed chunk shapes and premature termination before tool execution.
   by sandrogauci

2. 🟣 #197 bug: every tool call renders as agent_tool to OpenAI-compatible clients
   This is another OpenAI-compatible tool-call rendering/normalization bug in the gateway path. While the specific symptom differs, it confirms related problems in how function calls are converted for OpenAI-compatible clients and helps triangulate the tool-call handling area.
   by penradioGables

---

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:5311954574 --> @owui-terminator[bot] commented on GitHub (Aug 17, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟣 [#96](https://github.com/open-webui/computer/issues/96) **bug: OpenAI-compatible streaming usage can end responses early** *This reports a related OpenAI-compatible streaming bug where the implementation treats an intermediate event as final and ends the response too early. The new issue also centers on incorrect assumptions about streamed chunk shapes and premature termination before tool execution.* *by sandrogauci* 2. 🟣 [#197](https://github.com/open-webui/computer/issues/197) **bug: every tool call renders as `agent_tool` to OpenAI-compatible clients** *This is another OpenAI-compatible tool-call rendering/normalization bug in the gateway path. While the specific symptom differs, it confirms related problems in how function calls are converted for OpenAI-compatible clients and helps triangulate the tool-call handling area.* *by penradioGables* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

release

v0.9.21

v0.9.20

v0.9.19

v0.9.18

v0.9.17

v0.9.16

v0.9.15

v0.9.14

v0.9.13

v0.9.12

v0.9.11

v0.9.10

v0.9.9

v0.9.8

v0.9.7

v0.9.6

v0.9.5

v0.9.4

v0.9.3

v0.9.2

v0.9.1

v0.9.0

v0.8.8

v0.8.7

v0.8.6

v0.8.5

v0.8.4

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.7

v0.7.6

v0.7.5

v0.7.4

v0.7.3

v0.7.2

v0.7.1

v0.7.0

v0.6.2

v0.6.1

v0.6.0

v0.5.6

v0.5.5

v0.5.4

v0.5.3

v0.5.2

v0.5.1

v0.5.0

v0.4.10

v0.4.9

v0.4.8

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.4.0

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.9

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/computer#137

No description provided.