API: Ignore add_bos_token in chat completions

When fetching special tokens from the model, don't factor in the
add_bos_token and ban_eos_token parameters as switches.

In addition, change the internal handling of add_bos_token to an optional
boolean. This allows us to fallback to the model when selecting whether
or not to add the BOS token, especially for chat completions.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri
2025-05-01 22:51:15 -04:00
parent 3960612d38
commit aa657fa6e9
4 changed files with 13 additions and 17 deletions

View File

@@ -1,4 +1,4 @@
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, field_validator
from pydantic.json_schema import SkipJsonSchema
from time import time
from typing import Literal, Union, List, Optional, Dict
@@ -82,6 +82,12 @@ class ChatCompletionRequest(CommonCompletionRequest):
tool_call_end: SkipJsonSchema[Optional[str]] = None
tool_call_schema: SkipJsonSchema[Optional[dict]] = tool_call_schema
@field_validator("add_bos_token", mode="after")
def force_bos_token(cls, v):
"""Always disable add_bos_token with chat completions."""
return None
class ChatCompletionResponse(BaseModel):
id: str = Field(default_factory=lambda: f"chatcmpl-{uuid4().hex}")