* refac: use MilvusClient instead of deprecated ORM-style PyMilvus APIs
PyMilvus 2.6 emits a PyMilvusDeprecationWarning for every ORM-style call (`connections.connect`, `utility.*`, `Collection` and its methods) and will remove those APIs in PyMilvus 3.1. Both Milvus backends still used them, so a running instance floods its logs with deprecation warnings during indexing and retrieval, and would break outright once PyMilvus 3.1 lands.
Both vector clients now go through `MilvusClient`:
- `milvus_multitenancy.py`: collection creation, index creation, has_collection, insert, search, query iteration, delete and reset.
- `milvus.py`: the remaining ORM calls in `query()` (`connections.connect`, `Collection(...).load()`, `Collection.query_iterator`), plus the now-unused `FieldSchema` import.
Behaviour is unchanged: same schema, same index parameters and the same two-step scalar-index fallback, same filter expressions, same result shapes. Verified against embedded Milvus (milvus-lite, pymilvus 2.6.14) with a functional harness over both clients: insert, get, query by string/int/bool metadata filters, vector search, tenant isolation, oversized-text truncation, delete by id and by filter, delete_collection and reset all return identical results before and after, while the deprecation warnings drop from 57 to 0 for the multi-tenancy client and from 16 to 0 for the standard one.
One Milvus Lite nuance worth recording: `MilvusClient` sends index build parameters (`M`, `efConstruction`, `nlist`) as flat keys rather than as a nested `params` blob. A Milvus server accepts both forms, Milvus Lite only reads the nested one, so those tuning values are ignored on Lite. `MilvusClient` offers no way to send the nested form, and `milvus.py` already built its index parameters this way, so both backends are now consistent.
Fixes#26978
* refac: correct the Milvus scalar-index comment
The comment claimed that embedded Milvus Lite requires an explicit scalar index type. It does not: Milvus Lite rejects `create_index` on a VARCHAR field outright ("create_index only supports vector fields"), for every index type and with or without a metric type, so neither the parameterless call nor the explicit INVERTED fallback can succeed there. Filtered queries on `resource_id` still work on Lite, just unindexed.
Only the accurate half is kept, which is the reason the parameterless call is deliberate rather than an omission.
Enabling ENABLE_MILVUS_MULTITENANCY_MODE with the default MILVUS_URI (embedded Milvus Lite at DATA_DIR/vector_db/milvus.db) fails on the first embedding write: _create_shared_collection calls collection.create_index(RESOURCE_ID_FIELD) with no index params. A Milvus server auto-selects a scalar index type in that case, but Milvus Lite rejects the call with "create_index missing required 'index_type' parameter", so shared collection creation raises and every embedding write 500s (memory add, file upload, knowledge writes).
Keep the parameterless call as the first attempt so behavior on Milvus servers is unchanged, fall back to an explicit INVERTED scalar index, and if that also fails log a warning and continue. The scalar index only accelerates resource_id filters; inserts and filtered queries work without it, so a missing index must not break collection creation.
Verified against embedded Milvus Lite: shared collections now create (with the warning), and memory add, file upload and memory query succeed end to end. Against a Milvus server the first attempt is identical to the current code, so nothing changes where it works today.
has_collection did `collection_name in self.client.list_collections()`,
but chromadb's list_collections() returns Collection objects (1.x), not
name strings — so the membership test is always False, even when the
collection exists. Compare against the collection names instead (with a
hasattr guard tolerating versions that yield plain names).
Found via the dependency-contract test suite (unit/deps/test_chromadb.py).
The Qdrant backends implemented delete(ids=...) as a payload filter on
metadata.id, but points are stored with the item id as the Qdrant point id
(see _create_points), and not every point carries an id in its payload.
Memory points store only {created_at} in metadata (KB metadata embeddings
likewise), so deleting a single memory matched nothing and left an orphaned
vector that kept being injected into RAG context.
Delete by point id instead: PointIdsList for the standard backend, and a
tenant-scoped HasIdCondition for multitenancy (point ids are unique, so tenant
isolation is preserved). Filter-based deletion is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add support for Valkey vector database
Signed-off-by: Riley Des <riley.desserre@improving.com>
* feat: add CLIENT SETNAME to Valkey vector store connections
Set client_name on GlideClientConfiguration for both the main client
and batch client so connections are identifiable in CLIENT LIST,
monitoring dashboards, and CloudWatch metrics.
Signed-off-by: Riley Des <riley.desserre@improving.com>
---------
Signed-off-by: Riley Des <riley.desserre@improving.com>
Open WebUI's collection ACL accepted any unknown name as a
legacy/ephemeral collection. In Milvus multi-tenancy mode that name
becomes the `resource_id` and is interpolated unescaped into a SQL-like
Milvus expression — `resource_id == '<name>'` — so a name like
x' or resource_id != '' or resource_id == 'x
turns the filter into a tautology and returns every tenant's chunks
from the shared collection.
All collection names Open WebUI generates are UUIDs, SHA-256 hex
digests, or fixed-prefix variants of those — they all fit
[A-Za-z0-9_-]. Add a strict format check in
filter_accessible_collections (utils.py) that drops any name outside
that set before any ACL or vector-store lookup, applied even on the
admin bypass path. _validate_collection_access then surfaces the dropped
name as a 403.
As defense in depth, MilvusClient now validates resource_id at every
expression-construction site and escapes single quotes / backslashes in
any other string interpolated into a filter (delete ids, metadata
filter values). Non-string filter values are typed-checked instead of
str()-formatted.
Co-authored-by: Claude <noreply@anthropic.com>
Replace bare except clauses with except Exception to follow Python best practices and avoid catching unexpected system exceptions like KeyboardInterrupt and SystemExit.
In opensearch-py >= 3.0.0, IndicesClient.refresh() no longer accepts the
index name as a positional argument. This causes a TypeError when
uploading documents to knowledge bases with OpenSearch backend.
Changes positional arguments to keyword arguments (index=...) in all
three refresh() calls in the OpenSearch vector DB client.
Fixes#20649
* fix: Use get_index() instead of list_indexes() in has_collection() to handle pagination
Fixes#19233
Replace list_indexes() pagination scan with direct get_index() lookup
in has_collection() method. The previous implementation only checked
the first ~1,000 indexes due to unhandled pagination, causing RAG
queries to fail for indexes beyond the first page.
Benefits:
- Handles buckets with any number of indexes (no pagination needed)
- ~8x faster (0.19s vs 1.53s in testing)
- Proper exception handling for ResourceNotFoundException
- Scales to millions of indexes
* Update s3vector.py
Unneeded exception handling removed to match original OWUI code
* Adding hnsw index type for pgvector, allowing vector dimensions larger than 2000
* remove some variable assignments
* Make USE_HALFVEC variable configurable
* Simplify USE_HALFVEC handling
* Raise runtime error if the index requires rebuilt
---------
Co-authored-by: Moritz <moritz.mueller2@tu-dresden.de>
This commit introduces support for the DISKANN index type in the Milvus vector database integration.
Changes include:
- Added `MILVUS_DISKANN_MAX_DEGREE` and `MILVUS_DISKANN_SEARCH_LIST_SIZE` configuration variables.
- Updated the Milvus client to recognize and configure the DISKANN index type during collection creation.
The pymilvus library expects -1 for unlimited queries, but the code was passing None, which caused a TypeError. This commit changes the default value of the limit parameter in the query method from None to -1. It also updates the call site in the get method to pass -1 instead of None and updates the type hint and a comment to reflect this change.
Classic298