perf: halve Redis round trips on model resolution and socket pools (#27225)
When WEBSOCKET_MANAGER=redis, app.state.MODELS and the socket session/
usage pools are Redis-backed dicts, so every membership test and
getitem is a network round trip:
- generate_chat_completion checked `model_id not in models` (HEXISTS)
and then read `models[model_id]` (HGET) on every chat completion.
A single .get() now serves both, with the same not-found error.
- The direct-connection branch spread the pool with `{**MODELS, ...}`,
which iterates keys() then fetches each value — HKEYS plus one HGET
per model. dict(MODELS.items()) issues a single HGETALL instead.
- get_user_ids_from_room called SESSION_POOL.get(sid) twice per
session (once to filter, once for the value); the usage handler
checked membership then fetched the same key. Both now do one
lookup.
In non-Redis mode these are plain dicts and behavior is identical.
Claude-Session: https://claude.ai/code/session_01MHg5zs1VBjvRWQ54qHpfYD
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -179,8 +179,10 @@ async def generate_chat_completion(
|
||||
# Merge the direct connection model into server models so that
|
||||
# task functions (title, tags, etc.) can resolve a server-side
|
||||
# task model while still having the direct model available.
|
||||
# dict(...items()) is one HGETALL on a Redis-backed pool; ``{**pool}``
|
||||
# would issue HKEYS plus one HGET per model.
|
||||
models = {
|
||||
**request.app.state.MODELS,
|
||||
**dict(request.app.state.MODELS.items()),
|
||||
request.state.model['id']: request.state.model,
|
||||
}
|
||||
log.debug(f'direct connection to model: {request.state.model["id"]}')
|
||||
@@ -188,11 +190,12 @@ async def generate_chat_completion(
|
||||
models = request.app.state.MODELS
|
||||
|
||||
model_id = form_data['model']
|
||||
if model_id not in models:
|
||||
# Single lookup — membership check plus getitem would be two Redis
|
||||
# round trips on a Redis-backed model pool.
|
||||
model = models.get(model_id)
|
||||
if model is None:
|
||||
raise Exception('Model not found')
|
||||
|
||||
model = models[model_id]
|
||||
|
||||
if getattr(request.state, 'direct', False) and model_id == getattr(request.state, 'model', {}).get('id'):
|
||||
return await generate_direct_chat_completion(request, form_data, user=user, models=models)
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user