bhetherman/open-webui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: honor Admin UI web loader settings in get_web_loader (#26749)

Browse Source 
Since the config refactor, get_web_loader dispatched on the WEB_LOADER_ENGINE module constant, which is read from the environment once at import time. The engine selected in the Admin UI is stored under web.loader.engine in the config table but was never consulted, so UI-configured loader engines (external, playwright, firecrawl, tavily, microsoft_web_iq) were silently ignored and the built-in SafeWebBaseLoader always fetched pages directly. The same applied to the per-engine settings such as the external web loader URL and API key. This breaks egress-restricted deployments that rely on an external web loader: pages are fetched directly from the container and fail with errors like "Network is unreachable" even though an external loader is configured.

Pass the DB-backed loader settings into get_web_loader from both call sites, web search in process_web_search and web fetch via get_loader, and resolve every engine setting from them, keeping the module-level env constants as the fallback for keys that were never saved. Also initialise WebLoaderClass so an unknown engine raises the intended ValueError instead of an UnboundLocalError.

Fixes #26747
This commit is contained in:
Classic298
2026-07-24 01:30:47 -05:00
committed by GitHub
parent 18719fef9c
commit f7e7f32102
3 changed files with 56 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/open_webui/retrieval/utils.py
+15
View File
@@ -73,6 +73,20 @@ LOADER_CONFIG_KEYS = {
'web_loader_ssl_verification': 'web.loader.ssl_verification',
'web_loader_concurrent_requests': 'web.loader.concurrent_requests',
'web_search_trust_env': 'web.search.trust_env',
'web_loader_engine': 'web.loader.engine',
'web_loader_timeout': 'web.loader.timeout',
'playwright_ws_url': 'web.loader.playwright_ws_url',
'playwright_timeout': 'web.loader.playwright_timeout',
'firecrawl_api_key': 'web.loader.firecrawl_api_key',
'firecrawl_api_url': 'web.loader.firecrawl_api_url',
'firecrawl_timeout': 'web.loader.firecrawl_timeout',
'tavily_api_key': 'web.search.tavily_api_key',
'tavily_extract_depth': 'web.search.tavily_extract_depth',
'microsoft_web_iq_api_base_url': 'web.search.microsoft_web_iq_api_base_url',
'microsoft_web_iq_api_key': 'web.search.microsoft_web_iq_api_key',
'microsoft_web_iq_language': 'web.search.microsoft_web_iq_language',
'external_web_loader_url': 'web.loader.external_web_loader_url',
'external_web_loader_api_key': 'web.loader.external_web_loader_api_key',
'CONTENT_EXTRACTION_ENGINE': 'rag.content_extraction_engine',
'DATALAB_MARKER_API_KEY': 'rag.datalab_marker_api_key',
'DATALAB_MARKER_API_BASE_URL': 'rag.datalab_marker_api_base_url',
@@ -128,6 +142,7 @@ def get_loader(request, url: str, config: dict):
verify_ssl=config.get('web_loader_ssl_verification'),
requests_per_second=config.get('web_loader_concurrent_requests'),
trust_env=config.get('web_search_trust_env'),
loader_config=config,
)