fix: only route PDFs and images to the PaddleOCR-VL loader (#27529)

When `RAG_DOCUMENT_LOADER_ENGINE` is set to `paddleocr_vl`, the dispatch branch in `Loader._get_loader` checked only the engine name and a non-empty token, so every uploaded file was handed to the PaddleOCR-VL loader regardless of its type. Text based uploads such as `.md`, `.txt` and `.csv` were base64 encoded and posted to the `/layout-parsing` endpoint tagged as PDFs, and the API rejected them with `422 Unprocessable Entity` ("PDFium: Data format error"), so those files never indexed at all.

The loader already knows which extensions it can handle: it tags images with `fileType: 1` and treats everything else as a PDF. That list is now a module level constant, and the dispatch branch gates on `['pdf'] + images`, the same way `mistral_ocr`, `datalab_marker`, `document_intelligence` and `mineru` already limit themselves. Deriving the gate from the loader's own list keeps the two in sync, so a file can never be admitted by the gate and then mislabelled as a PDF on the wire. Everything outside that set falls through to the default loader chain, so `.md` and `.txt` load as text, `.csv` through `CSVLoader`, `.docx` through `Docx2txtLoader`, and so on.

The branch also never checked `PADDLEOCR_VL_BASE_URL`. With the URL cleared, `PaddleOCRVLLoader` raised `ValueError` from its constructor and the upload failed outright instead of falling back. Both settings are now required for the branch to be taken, matching how the other engines guard their own configuration.

Fixes #24988
Fixes #26759
This commit is contained in:
Classic298
2026-07-26 17:34:59 -04:00
committed by GitHub
parent 4ac22b89fd
commit 225e238856
2 changed files with 11 additions and 4 deletions
+7 -2
View File
@@ -27,7 +27,7 @@ from open_webui.retrieval.loaders.datalab_marker import DatalabMarkerLoader
from open_webui.retrieval.loaders.external_document import ExternalDocumentLoader
from open_webui.retrieval.loaders.mineru import MinerULoader
from open_webui.retrieval.loaders.mistral import MistralLoader
from open_webui.retrieval.loaders.paddleocr_vl import PaddleOCRVLLoader
from open_webui.retrieval.loaders.paddleocr_vl import PADDLEOCR_VL_SUPPORTED_EXTENSIONS, PaddleOCRVLLoader
logging.basicConfig(stream=sys.stdout, level=GLOBAL_LOG_LEVEL)
log = logging.getLogger(__name__)
@@ -556,7 +556,12 @@ class Loader:
use_base64=self.kwargs.get('MISTRAL_OCR_USE_BASE64', False),
user=self.user,
)
elif self.engine == 'paddleocr_vl' and self.kwargs.get('PADDLEOCR_VL_TOKEN') != '':
elif (
self.engine == 'paddleocr_vl'
and self.kwargs.get('PADDLEOCR_VL_BASE_URL')
and self.kwargs.get('PADDLEOCR_VL_TOKEN')
and file_ext in PADDLEOCR_VL_SUPPORTED_EXTENSIONS
):
loader = PaddleOCRVLLoader(
api_url=self.kwargs.get('PADDLEOCR_VL_BASE_URL'),
token=self.kwargs.get('PADDLEOCR_VL_TOKEN'),
@@ -11,6 +11,9 @@ from open_webui.env import GLOBAL_LOG_LEVEL
logging.basicConfig(stream=sys.stdout, level=GLOBAL_LOG_LEVEL)
log = logging.getLogger(__name__)
PADDLEOCR_VL_IMAGE_EXTENSIONS = ['png', 'jpg', 'jpeg', 'bmp', 'tiff', 'webp']
PADDLEOCR_VL_SUPPORTED_EXTENSIONS = ['pdf'] + PADDLEOCR_VL_IMAGE_EXTENSIONS
class PaddleOCRVLLoader:
"""Loader that uses PaddleOCR-vl API to extract text from PDF/images."""
@@ -46,8 +49,7 @@ class PaddleOCRVLLoader:
# Detect fileType based on file extension
ext = self.file_path.lower().split('.')[-1]
image_extensions = ['png', 'jpg', 'jpeg', 'bmp', 'tiff', 'webp']
file_type = 1 if ext in image_extensions else 0
file_type = 1 if ext in PADDLEOCR_VL_IMAGE_EXTENSIONS else 0
payload = {
'file': file_data,