Match WEB_FETCH_FILTER_LIST on hostnames with label boundaries, not URL suffix (CWE-693) (#25949)

is_string_allowed does endswith() matching and was called with the full URL
(retrieval/web/utils.py) against WEB_FETCH_FILTER_LIST, so a blocklisted host with any
path (https://blocked.example/x) ended with /x, not the host, and slipped through; the
allowlist direction false-rejected legitimate URLs and admitted attacker URLs ending in
an allowed string. The same endswith caused label confusion at the hostname call site
(retrieval/web/main.py): corp.com matched evilcorp.com, 10.0.0.1 matched 110.0.0.1.

Add is_host_allowed(host, ...) matching on DNS label boundaries (host == pattern or
host.endswith('.' + pattern)), called with the parsed hostname at both web-fetch call
sites. is_string_allowed is left unchanged for the unrelated function-name filters
(utils/middleware.py, utils/tools.py).

The separate is_global guard (validate_url / _ssrf_safe_new_conn, active when
ENABLE_RAG_LOCAL_WEB_FETCH is off) already blocks RFC1918/loopback/link-local, so this
restores the admin's intended blocking of specific public hosts.

Co-authored-by: addcontent <59762500+addcontent@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

backend/open_webui/retrieval/web/utils.py

@@ -55,7 +55,7 @@ from open_webui.env import (
 from open_webui.retrieval.loaders.external_web import ExternalWebLoader
 from open_webui.retrieval.loaders.tavily import TavilyLoader
 from open_webui.retrieval.web.firecrawl import scrape_firecrawl_url
-from open_webui.utils.misc import is_string_allowed
+from open_webui.utils.misc import is_host_allowed
 
 log = logging.getLogger(__name__)
 
@@ -93,7 +93,9 @@ def validate_url(url: Union[str, Sequence[str]]):
 
         # Blocklist check using unified filtering logic
         if WEB_FETCH_FILTER_LIST:
-            if not is_string_allowed(url, WEB_FETCH_FILTER_LIST):
+            # Match on the parsed hostname, not the full URL: a path component would
+            # otherwise let any URL slip past a hostname-based block/allow entry.
+            if not is_host_allowed(parsed_url.hostname, WEB_FETCH_FILTER_LIST):
                 log.warning(f'URL blocked by filter list: {url}')
                 raise ValueError(ERROR_MESSAGES.INVALID_URL)