perf: linear content accumulation in the streaming response handler (#27231)
The streaming handler rebuilt the full accumulated response with
`content = f'{content}{value}'` on every content delta — a complete
string copy per chunk, making accumulation O(n^2) over the response
length. Use in-place `content += value` for the (universal) str case,
which CPython extends in place, keeping accumulation O(n); the
f-string fallback is preserved for non-str values so coercion
behavior is unchanged.
In the ENABLE_REALTIME_CHAT_SAVE branch, full_output() — which
concatenates the entire accumulated output — was called twice per
chunk (once for the DB upsert, once for the emitted delta). Compute
it once and reuse.
Claude-Session: https://claude.ai/code/session_01MHg5zs1VBjvRWQ54qHpfYD
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -4490,7 +4490,12 @@ async def streaming_chat_response_handler(response, ctx):
|
||||
user,
|
||||
)
|
||||
|
||||
content = f'{content}{value}'
|
||||
if isinstance(value, str):
|
||||
# In-place append — avoids copying the full
|
||||
# accumulated response on every chunk.
|
||||
content += value
|
||||
else:
|
||||
content = f'{content}{value}'
|
||||
|
||||
# Check if we're inside a tag-based block
|
||||
# (reasoning, code_interpreter, or solution).
|
||||
@@ -4597,16 +4602,17 @@ async def streaming_chat_response_handler(response, ctx):
|
||||
if ENABLE_REALTIME_CHAT_SAVE and not metadata.get('chat_id', '').startswith(
|
||||
'channel:'
|
||||
):
|
||||
current_output = full_output()
|
||||
# Save message in the database
|
||||
await Chats.upsert_message_to_chat_by_id_and_message_id(
|
||||
metadata['chat_id'],
|
||||
metadata['message_id'],
|
||||
{
|
||||
'output': full_output(),
|
||||
'output': current_output,
|
||||
},
|
||||
)
|
||||
data = {
|
||||
'output': full_output(),
|
||||
'output': current_output,
|
||||
}
|
||||
delta_type = 'content'
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user