Commit Graph
58 Commits
Author SHA1 Message Date
e7ba8978c6 fix: reject parser-confusing chars in validate_url to close SSRF bypass (#24534)
urllib.parse.urlparse and requests/aiohttp disagree on how to split URLs
containing backslash, tab, CR, or LF in or around the netloc. urlparse
treats backslash as part of userinfo and uses what follows '@' as the
host; requests treats backslash as the start of the path and connects
to whatever precedes it. The same URL therefore passes the private-IP
filter (urlparse sees a public host) but reaches an internal target
(requests connects to e.g. 127.0.0.1). End result is an SSRF that the
existing IP block list cannot catch because it's evaluating the wrong
host.

PoC: http://127.0.0.1:6666\@1.1.1.1 — urlparse hostname is 1.1.1.1
(global, passes), requests reaches 127.0.0.1 (loopback).

Reject up front any URL containing one of the four documented parser-
confusing characters before either parser gets a chance to interpret
it. None of these characters is valid in an unencoded URL (\ should
always be %5C, whitespace should be %09 / %0A / %0D), so this is a
pure defensive rejection with no legitimate-input false positives.

Reported by Fushuling and RacerZ-fighting in GHSA-8w7q-q5jp-jvgx.

Co-authored-by: Fushuling <Fushuling@users.noreply.github.com>
Co-authored-by: RacerZ-fighting <RacerZ-fighting@users.noreply.github.com>
2026-05-11 00:57:48 +09:00
Timothy Jaeryang Baek df42d96c95 refac 2026-05-09 21:05:49 +09:00
Classic298 8854541508 fix: prevent redirect-based SSRF in web-fetch and image-load call sites (#24491)
validate_url() in retrieval/web/utils.py only validates the initial URL.
The HTTP clients used downstream (sync requests, sync requests via the
parent WebBaseLoader._scrape, aiohttp via load_url_image) followed 3xx
redirects by default and did not re-validate the redirect target against
the private-IP / metadata-IP block list. An authenticated user could
submit a public URL that 302-redirected to an internal address (RFC1918,
127.0.0.1, 169.254.169.254, etc.) and the redirected response was returned
to them, enabling SSRF reads of internal services and cloud metadata.

Three call sites needed allow_redirects=False to match the policy already
enforced on the async _fetch() path:

- SafeWebBaseLoader: override requests_kwargs in __init__ so that the
  inherited synchronous _scrape() path passes allow_redirects=False to
  self.session.get() (the parent WebBaseLoader uses requests' default
  allow_redirects=True).
- get_content_from_url (retrieval/utils.py): pass allow_redirects=False
  on the streamed requests.get(...) call.
- load_url_image (routers/images.py, image-edits endpoint): pass
  allow_redirects=False on the aiohttp session.get(...) call.

Reports consolidated under GHSA-rh5x-h6pp-cjj6:
- GHSA-rh5x-h6pp-cjj6 (tenbbughunters / Tenable) - sync _scrape
- GHSA-5vxg-6gmv-m2qr (YLChen-007) - load_url_image
- GHSA-hf76-c83f-63w2 (tempcollab) - aiohttp _fetch (already fixed)
- GHSA-h55f-h5fh-mvm4 (sneaXOR) - get_content_from_url
2026-05-09 21:01:45 +09:00
RomualdYTandTim Baek e0d6074cd2 refactor(firecrawl): use v2 API directly (#23934)
Co-authored-by: Tim Baek <tim@openwebui.com>
2026-04-24 18:32:08 +09:00
Timothy Jaeryang Baek 0e311a95a7 refac 2026-04-24 15:16:37 +09:00
Timothy Jaeryang Baek fd25152076 refac 2026-04-20 08:34:15 +09:00
Timothy Jaeryang Baek 9c64d84ad9 refac 2026-04-13 15:03:22 -05:00
Classic298 0753409e7b fix: use ipaddress stdlib for IPv6 SSRF protection (#23453)
The validators.ipv6(ip, private=True) call always returns a falsy ValidationError because validators==0.35.0 does not support the private kwarg for IPv6. This means any hostname resolving to a private IPv6 address (::1, fd00::*, ::ffff:169.254.169.254) bypasses SSRF protection entirely, circumventing the fix for CVE-2025-65958.

Replace both the IPv4 and IPv6 validators-based private checks with Python's stdlib ipaddress module using an allowlist approach (not addr.is_global). This blocks all non-globally-routable addresses — private, loopback, link-local, reserved, multicast, and unspecified — for both IPv4 and IPv6, including IPv4-mapped IPv6 addresses.
2026-04-12 16:34:13 -05:00
Timothy Jaeryang Baek de3317e26b refac 2026-03-17 17:58:01 -05:00
Tim Baek a214ec40ea fix 2026-02-06 03:34:21 +04:00
Timothy Jaeryang Baek 89ad1c68d1 enh: FIRECRAWL_TIMEOUT 2026-01-01 02:07:22 +04:00
823b9a6dd9 chore/perf: Remove old SRC level log env vars with no impact (#20045)
* Update openai.py

* Update env.py

* Merge pull request open-webui#19030 from open-webui/dev (#119)

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-20 08:16:14 -05:00
Timothy Jaeryang Baek b02397e460 feat: WEB_LOADER_TIMEOUT 2025-12-08 11:49:27 -05:00
Timothy Jaeryang Baek 743199f2d0 feat/enh: tool server function name filter list 2025-11-25 02:31:34 -05:00
Timothy Jaeryang Baek 2af4c4b3c7 refac 2025-11-18 04:42:09 -05:00
Timothy Jaeryang BaekandClassic298 02238d3113 feat/security: Add SSRF protection with configurable blocklist
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2025-11-18 04:40:55 -05:00
Timothy Jaeryang Baek 7faf19dad9 refac 2025-11-06 15:21:06 -05:00
Omar Aburub 3bcf9a442a fix: make SSL verification async 2025-10-29 16:14:53 +03:00
wei840222 7a3f4d85f6 refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader 2025-10-26 15:05:35 +08:00
Timothy Jaeryang Baek e000494e48 refac 2025-10-07 11:53:30 -05:00
Timothy Jaeryang Baek 0214c1e66c refac 2025-09-09 16:48:59 +04:00
_00_ 093af754e7 FIX: Playwright Timeout (ms) interpreted as seconds
Fix for Playwright Timeout (ms) interpreted as seconds.

To address https://github.com/open-webui/open-webui/issues/16801

In Frontend Playwright Timeout is setted as (ms), but in backend is interpreted as (s) doing a time conversion for playwright_timeout var (that have to be in ms).

& as  _Originally posted by @rawbby in [#16801](https://github.com/open-webui/open-webui/issues/16801#issuecomment-3216782565)_

> I personally think milliseconds are a reasonable choice for the timeout. Maybe the conversion should be fixed, not the label.
> This would further not break existing configurations from users that rely on their current config.
>
2025-08-23 14:15:00 +02:00
tth37 78befd5a2f fix: Default web loader fail when verify_ssl=False 2025-05-20 19:44:18 +08:00
Timothy Jaeryang Baek b143c71da2 refac: AIOHTTP_CLIENT_SESSION_SSL 2025-05-14 23:33:52 +04:00
Timothy Jaeryang Baek 42382b5167 fix 2025-05-14 22:46:01 +04:00
Timothy Jaeryang Baek 8732b64b6b feat: external document loader support 2025-05-14 22:28:40 +04:00
tth37 8f7195ceda fix: FireCrawlLoader default mode to scrape 2025-04-24 01:17:35 +08:00
Timothy Jaeryang Baek 09874ab83d fix: FireCrawlLoader 2025-04-24 01:40:34 +09:00
tth37 839ba22c90 feat: Backend for Self-Hosted/External Web Search/Loader Engines 2025-04-14 01:49:05 +08:00
Timothy Jaeryang Baek 48a23ce3fe refac: web/rag config 2025-04-12 16:33:36 -07:00
Timothy Jaeryang Baek 9b20ef4922 refac 2025-03-20 14:01:47 -07:00
genjuro 07098c6352 perf: set shorter timeout for playwright and make it configurable 2025-03-20 15:28:09 +08:00
leilibj 3e8546135d fix: correct incorrect usage of log.exception method 2025-03-19 13:04:34 +08:00
orenzhang c761e4fd08 feat(trace): opentelemetry instrument 2025-03-10 22:27:31 +08:00
Luke 7917128ed3 enh: enable configuration for tavily extract depth 2025-03-08 00:43:02 -05:00
Luke 987954c817 feat: Add Tavily extract web loader integration 2025-03-06 18:15:18 -05:00
Timothy Jaeryang Baek eeb00a5ca2 chore: format 2025-02-20 01:01:29 -08:00
Youggls 0fb3c08181 feat: Add Firecrawl web loader integration 2025-02-19 16:54:44 +08:00
Rory 10e0c81de9 Merge remote-tracking branch 'upstream/dev' into playwright
# Conflicts:
#	backend/open_webui/retrieval/web/utils.py
#	backend/open_webui/routers/retrieval.py
2025-02-17 21:53:39 -06:00
Rory bc82f48ebf refac: RAG_WEB_LOADER -> RAG_WEB_LOADER_ENGINE 2025-02-17 21:43:32 -06:00
Craig Quiter e67eb89e05 style: black format 2025-02-15 10:53:16 -08:00
Rory 8e9b00a017 Fix docstring 2025-02-14 22:48:15 -06:00
Rory aa2b764d74 Finalize incomplete merge to update playwright branch
Introduced feature parity for trust_env
2025-02-14 22:32:45 -06:00
Rory 4da220c513 Merge remote-tracking branch 'upstream/dev' into playwright
# Conflicts:
#	backend/open_webui/config.py
#	backend/open_webui/main.py
#	backend/open_webui/retrieval/web/utils.py
#	backend/open_webui/routers/retrieval.py
#	backend/open_webui/utils/middleware.py
#	pyproject.toml
2025-02-14 20:48:22 -06:00
Guofeng Yi b38acc8559 Merge branch 'dev' into feate-webloader-support-proxy 2025-02-15 09:50:02 +08:00
Yimi81 d3f71930f0 web loader support proxy 2025-02-14 07:15:09 +00:00
Yimi81 ceef600223 support async load for websearch 2025-02-14 07:05:10 +00:00
Rory ec6fe9939b Merge remote-tracking branch 'upstream/dev' into playwright 2025-02-05 17:47:58 -06:00
Timothy Jaeryang Baek 3adfa29f7d chore: format 2025-02-03 21:56:35 -08:00
Rory 7bac1a170d Merge remote-tracking branch 'upstream/dev' into playwright
# Conflicts:
#	backend/open_webui/retrieval/web/utils.py
2025-02-03 22:32:46 -06:00