Every page pulled in by web search and web RAG is parsed with BeautifulSoup's `html.parser`, a pure-Python parser. It is the slowest option bs4 offers, and it is being handed 300 KiB to 1.5 MiB documents, several per query. `SafeWebBaseLoader` inherits `default_parser = "html.parser"` from langchain's `WebBaseLoader` and never overrides it, so this is an upstream default carried by accident, not a decision anyone made for Open WebUI.
`default_parser` is the single chokepoint for both the sync `_scrape()` path and the async `ascrape_all()` path, so one `setdefault` covers everything and an explicit caller override still wins.
lxml is already in the tree as a transitive hard dependency of ddgs, python-pptx and unstructured, so nothing new enters the image and `uv.lock` already resolves it at 6.1.1. The pin makes it explicit and closes a latent failure: bs4's `"xml"` feature, already used for `.xml` URLs in `_unpack_fetch_results()`, requires lxml and would raise `FeatureNotFound` the day that transitive dependency moves.
## Benchmarks
37 real pages, 13.8 MiB of HTML, median of 5 runs each. The timed operation is `BeautifulSoup(html, parser)` plus `get_text()` plus `extract_metadata()`, which is exactly what the loader does per page. bs4 4.14.3, lxml 6.1.1, CPython 3.12.
| | html.parser | lxml | |
|---|---|---|---|
| 37 pages, 13.8 MiB total | 1611.0ms | 1151.3ms | 1.4x faster, 460ms saved |
Largest pages:
| page | size | html.parser | lxml | speedup |
|---|---|---|---|---|
| pypi.org/project/aiohttp/ | 1259 KiB | 243.75ms | 180.51ms | 1.4x |
| gnu.org/software/bash/manual/bash.html | 1017 KiB | 257.97ms | 178.99ms | 1.4x |
| rfc-editor.org/rfc/rfc9110.html | 1157 KiB | 205.94ms | 154.87ms | 1.3x |
| docs.aiohttp.org/en/stable/client_reference.html | 403 KiB | 108.62ms | 84.93ms | 1.3x |
| ollama.com/library | 779 KiB | 117.55ms | 73.64ms | 1.6x |
| theregister.com | 1052 KiB | 88.32ms | 60.12ms | 1.5x |
| kubernetes.io/docs/concepts/services-networking/service/ | 563 KiB | 72.18ms | 43.43ms | 1.7x |
| docs.python.org/3/library/socket.html | 301 KiB | 71.88ms | 49.04ms | 1.5x |
Ranges from 1.1x to 1.7x, and the win grows with page size. A ten result web search sheds roughly 125ms of parsing. Because the async path builds its soups inline in `_unpack_fetch_results()`, that is 125ms the event loop spends parsing HTML instead of serving other users' streams. Pages under about 10 KiB are marginally slower under lxml due to fixed setup cost, which is worth nothing either way.
## Output verification
The risk in changing parser is silently different extracted text, so that was measured rather than assumed. Across all 37 real pages:
- **Zero characters of text were lost.** Every diff opcode against html.parser output was an insertion. Not one page dropped content under lxml.
- 659 characters were added, all on one page (docs.docker.com), where an inline Alpine.js `@click` handler containing a regex confuses libxml2's attribute handling and leaks a 73-character JS fragment into the text nine times. That is 659 characters of script noise in 27,206 characters of extracted text, with no content affected.
- Metadata (`title`, `description`, `language`) was identical on 35 of 37 pages. The two exceptions are 141-byte Wikipedia bot-block stubs with no `<html>` element, where lxml's fragment auto-wrapping adds `language: "No language found."`. Both parsers extract the same text from them.
Large documents were checked separately because libxml2 carries internal size caps. A 12 MiB single text node, 12 MiB spread across 400k nodes, a 3 MiB attribute value and 50k sibling elements with a trailing marker all produced byte-identical text under both parsers, with no truncation.
Malformed markup was checked too. lxml and html.parser diverge on unterminated comments, bare CDATA and duplicated `<html>` elements, all cases where both parsers are guessing and neither is correct. None of those shapes appeared in the 37 page corpus.
`backend/open_webui/env.py:184` also uses `html.parser`, on the local CHANGELOG at import time. That is trivial input on a startup path and is deliberately left alone.
The migration to joserfc completed the job but left the old dependency pinned. `python-jose` now has zero imports anywhere in the backend: the only `jose` references left are `joserfc` in `utils/oauth.py`, and a repo-wide search for `from jose`, `import jose` or `python_jose` returns nothing outside the three pin files.
Removing it also removes `ecdsa` and `rsa` from the image, which were pulled in only by python-jose. `uv lock` confirms that: it drops exactly those three packages and nothing else, because google-auth 2.55 depends on cryptography and pyasn1-modules rather than rsa. That is worth having beyond the size saving, since `ecdsa` ships a documented Minerva-style timing side-channel in its P-256 signing path that upstream has declined to fix, so keeping it in the image means shipping a flagged crypto library that nothing calls.
Verified by blocking the `jose` module at import time and importing the backend anyway:
```
PASS import open_webui.utils.auth
PASS import open_webui.utils.oauth
PASS import open_webui.main
jose in sys.modules: False
PASS create_token/decode_token round trip
```
One user-visible consequence worth stating: Tools and Functions run in the same interpreter, so a third-party plugin that imports `jose` directly stops working after this. Nothing in Open WebUI itself does, and PyJWT remains a dependency, but a plugin relying on a library the application never declared for that purpose is the only thing this can break.
`uv.lock` was edited surgically rather than regenerated, to avoid the unrelated whole-file churn a newer uv version introduces. The result was diffed against real `uv lock` output and matches it exactly apart from that version's cosmetic fields.
Uvicorn's `--ws auto` selected its `websockets_impl` protocol on 0.41.0, which is built on `websockets.legacy`. That module raises `AssertionError` in `_drain_helper` during keepalive pings and kills the websocket connection. Each crash runs the Socket.IO `disconnect` handler and drops the session from `SESSION_POOL`, so every subsequent server-to-browser call fails. The most visible symptom is the Pyodide code execution tool, which reaches the browser through `sio.call('events', ...)` and returns `{"stderr": "Client session disconnected."}` on every run.
Uvicorn 0.50.0 changed `--ws auto` to select the sans-io implementation whenever websockets is installed, and deprecated the legacy one. Bumping the pin therefore fixes this on every launch path at once, without adding a `--ws` flag to the startup scripts. Doing nothing is not stable either: websockets is unpinned apart from uvicorn's own `>=13.0` floor, and `websockets.legacy` is removed outright in websockets 17, which turns the current AssertionError into an ImportError on a fresh install.
Bumping to 0.51.0 rather than the minimum 0.50.0 also picks up the sans-io keepalive pings added in 0.44.0, so raw websocket endpoints keep the idle-timeout behaviour they have today behind a reverse proxy. Uvicorn 0.51.0 drops colorama from its `standard` extra and raises the httptools floor to 0.8.0, which the lockfile already satisfies.
Verified on the bumped pin: the backend boots, `/health` returns 200, `--ws auto` resolves to `WebSocketsSansIOProtocol`, a Socket.IO client completes a websocket handshake against the running app, and a bidirectional `sio.call` round trip succeeds. The unit test suite reports an identical 2273 passed / 7 failed on 0.41.0 and 0.51.0, with the 7 failures unrelated to uvicorn.
Fixes#27550
* perf: cut per-instance CPU cost of shared socket.io Redis pub/sub channel
Profiling a multi-instance deployment (py-spy --gil) showed ~44% of worker
CPU in the socket.io pub/sub listener. Two causes, two fixes:
- Add hiredis so redis-py parses the RESP protocol in C instead of pure
Python (redis/_parsers/resp3.py alone accounted for ~28% of GIL samples;
redis-py auto-selects the hiredis parser when importable).
- Subclass AsyncRedisManager to drop emits whose target room has no local
participants before upstream _handle_emit re-encodes the full packet.
Every instance receives every emit published on the shared channel, so
with N instances all but the hosting one were paying full packet
re-serialization per message just to deliver it to nobody. Broadcasts
(room=None) are unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9CQ9qnp3sZGYQQwztsCJT
* fix: restrict pub/sub emit early-out to string rooms
Adversarial review against python-socketio 5.16.2 found one divergence
from upstream: for a degenerate empty-sequence room (emit to room=[]) on
an instance whose namespace has no local clients, the filter's
get_participants probe raises IndexError from room[0] where upstream
returns silently at the namespace guard and still publishes to Redis.
Open WebUI only ever emits to scalar string rooms or room=None, so the
case is unreachable today; guard on isinstance(room, str) anyway so any
non-string room shape passes through to upstream behavior unchanged.
Every open-webui emit uses a string room, so the fast path still covers
all real traffic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9CQ9qnp3sZGYQQwztsCJT
* Update requirements.txt
* Update pyproject.toml
* Update requirements-min.txt
---------
Co-authored-by: Claude <noreply@anthropic.com>
* chore: bump Python backend dependencies, drop unused peewee
Minor/patch + reviewed major bumps across requirements.txt,
requirements-min.txt, pyproject.toml and uv.lock; playwright image bumped in
docker-compose.playwright.yaml. peewee/peewee-migrate removed (zero imports).
Security-relevant: cryptography 46->48, authlib 1.6.10->1.7.2, PyJWT 2.11->2.13,
requests 2.33.1->2.34.2, RestrictedPython 8.1->8.2, pillow 12.1.1->12.2.0.
Reviewed majors: redis 7->8, pymilvus ->2.6.14, azure-search-documents 11->12,
chardet 5->7, unstructured 0.18->0.22, pycrdt 0.12->0.13.
Testing:
- Resolution: `uv lock` resolves the full bumped set with no conflicts; uv.lock
regenerated to match (peewee dropped, every pin including
azure-search-documents==12.0.0 resolves).
- Per-dependency contract tests (external tests repo, unit/deps/): 105 files,
2205 passed / 6 skipped, ruff-clean. One file per dependency pins the symbols,
signatures and behaviour the backend actually uses, so an API removal/rename in
a bumped version fails loudly instead of at runtime. Offline/deterministic.
- End-to-end embed->retrieve test driving transformers + sentence-transformers +
chromadb together through Open WebUI's real RAG path (cached model, in-memory
chroma, semantic retrieval asserted).
- Install/startup/health resolution gate added to the dep-bump workflow and the
integration suite (uv/pip resolve + uvicorn /health + Playwright dev visibility).
- Bugs surfaced while testing each got an isolated fix branch + regression test:
Mistral OCR aiohttp FilePayload (#25779), chroma has_collection (#25780),
aiocache per-user model-cache key (security), otel semconv deprecation,
pydub/audioop <3.13 note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bump python-multipart 0.0.22 -> 0.0.27 (CVE-2026-42561, CVE-2026-40347)
0.0.22 is affected by two DoS CVEs in the multipart parser that
Starlette/FastAPI run for every multipart/form-data request, so any
authenticated user hitting an upload endpoint can trigger them:
- CVE-2026-42561: unbounded part-header count/size -> CPU exhaustion (fixed 0.0.27)
- CVE-2026-40347: large multipart preamble/epilogue DoS (fixed 0.0.26)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add support for Valkey vector database
Signed-off-by: Riley Des <riley.desserre@improving.com>
* feat: add CLIENT SETNAME to Valkey vector store connections
Set client_name on GlideClientConfiguration for both the main client
and batch client so connections are identifiable in CLIENT LIST,
monitoring dashboards, and CloudWatch metrics.
Signed-off-by: Riley Des <riley.desserre@improving.com>
---------
Signed-off-by: Riley Des <riley.desserre@improving.com>
* chore(deps): update and synchronize backend dependencies
- Updated dependencies in requirements files and pyproject.toml to latest versions.
- Preserved pinned versions for av, pinecone, and pyarrow.
- Added missing dependencies to pyproject.toml to match requirements.txt.
- Ensured all dependency versions are synchronized across files.
* Update pyproject.toml
Timothy Jaeryang Baek