Enabling ENABLE_MILVUS_MULTITENANCY_MODE with the default MILVUS_URI (embedded Milvus Lite at DATA_DIR/vector_db/milvus.db) fails on the first embedding write: _create_shared_collection calls collection.create_index(RESOURCE_ID_FIELD) with no index params. A Milvus server auto-selects a scalar index type in that case, but Milvus Lite rejects the call with "create_index missing required 'index_type' parameter", so shared collection creation raises and every embedding write 500s (memory add, file upload, knowledge writes).
Keep the parameterless call as the first attempt so behavior on Milvus servers is unchanged, fall back to an explicit INVERTED scalar index, and if that also fails log a warning and continue. The scalar index only accelerates resource_id filters; inserts and filtered queries work without it, so a missing index must not break collection creation.
Verified against embedded Milvus Lite: shared collections now create (with the warning), and memory add, file upload and memory query succeed end to end. Against a Milvus server the first attempt is identical to the current code, so nothing changes where it works today.
Audit of asyncio.sleep vs time.sleep and event-loop-blocking calls:
- utils/plugin.py: run pip `install_frontmatter_requirements`
(subprocess.check_call) via asyncio.to_thread in load_tool_module_by_id,
load_function_module_by_id, and install_tool_and_function_dependencies.
- retrieval/utils.py: move the synchronous SSRF-guarded requests probe and
loader.load() in get_content_from_url into a sync helper run via
asyncio.to_thread.
- routers/audio.py: write uploaded audio to disk off the event loop in
transcription().
- routers/pipelines.py: write uploaded pipeline file off the event loop in
upload_pipeline().
The existing time.sleep call sites are all in genuinely synchronous
functions (sync requests/DB drivers/daemon threads) with async
counterparts that already use asyncio.sleep, so no time.sleep -> asyncio.sleep
changes were needed.
Claude-Session: https://claude.ai/code/session_01LXR5bYfsfSS42RGHQZu2Ta
Co-authored-by: Claude <noreply@anthropic.com>
Web search results stored as a vector collection were silently dropped
before retrieval when BYPASS_RETRIEVAL_ACCESS_CONTROL is False (the
default). The server-generated web_search file item carries a
'collection_name' but its 'web_search' type is not matched by any
explicit dispatch branch in get_sources_from_items, so it fell through
to the untrusted client-supplied collection_name branch and was ignored.
Add an explicit branch for type == 'web_search' items so the collection
is queried again. Access control is preserved: the collection still
passes through filter_accessible_collections, which already allowlists
web-search-* and bypasses only for admins.
Regression introduced when the retrieval access-control hardening gated
the bare collection_name fallback behind BYPASS_RETRIEVAL_ACCESS_CONTROL.
get_embedding_function runs at import time (main.py), so the empty-engine
+ no-model guard added in 55ca719b raised ValueError during startup and
bricked the instance: a blank embedding model set via the UI made the app
unbootable, with no way back into settings to undo it. Move the check into
the returned coroutine so construction always succeeds and the error only
fires when something actually embeds, surfacing as a clear request error
instead of the old cryptic 'NoneType has no attribute encode'.
Fixes#25634Fixes#25165
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The web-ingest probe in get_content_from_url validated the URL at resolve
time (validate_url) but then fetched with a bare requests.get, which
re-resolves the hostname at connect time. An attacker-controlled name
server can answer with a public IP during validation and an internal IP
at connect, reaching cloud metadata / loopback / internal services (blind
always; binary content-types are read back to the caller). The
connection-layer guard (#24759) that closes this for the SafeWebBaseLoader
path was never mounted on this probe.
Route the probe through the same _SSRFSafeAdapter the loader uses, so the
resolution that feeds the TCP connect is re-validated against the
global-IP check.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
has_collection did `collection_name in self.client.list_collections()`,
but chromadb's list_collections() returns Collection objects (1.x), not
name strings — so the membership test is always False, even when the
collection exists. Compare against the collection names instead (with a
hasattr guard tolerating versions that yield plain names).
Found via the dependency-contract test suite (unit/deps/test_chromadb.py).
_upload_file_async built its multipart body with
`aiohttp.streams.FilePayload(...)`, which no longer exists in aiohttp
(payload classes live in aiohttp.payload, and there is no FilePayload).
The reference sits inside a lazy closure, so import succeeds and only the
async Mistral OCR file-upload path blows up at runtime with
AttributeError on every call.
Mirror the working sync path: open the file in a context manager and let
MultipartWriter.append(file_obj, {...}) build a streaming
BufferedReaderPayload, with the POST issued inside the open() block so
the handle stays valid for the whole upload. Preserves the streaming /
memory-efficiency intent.
Found via the dependency-contract test suite (unit/deps/test_aiohttp.py),
which pins aiohttp.streams.FilePayload as absent.
is_string_allowed does endswith() matching and was called with the full URL
(retrieval/web/utils.py) against WEB_FETCH_FILTER_LIST, so a blocklisted host with any
path (https://blocked.example/x) ended with /x, not the host, and slipped through; the
allowlist direction false-rejected legitimate URLs and admitted attacker URLs ending in
an allowed string. The same endswith caused label confusion at the hostname call site
(retrieval/web/main.py): corp.com matched evilcorp.com, 10.0.0.1 matched 110.0.0.1.
Add is_host_allowed(host, ...) matching on DNS label boundaries (host == pattern or
host.endswith('.' + pattern)), called with the parsed hostname at both web-fetch call
sites. is_string_allowed is left unchanged for the unrelated function-name filters
(utils/middleware.py, utils/tools.py).
The separate is_global guard (validate_url / _ssrf_safe_new_conn, active when
ENABLE_RAG_LOCAL_WEB_FETCH is off) already blocks RFC1918/loopback/link-local, so this
restores the admin's intended blocking of specific public hosts.
Co-authored-by: addcontent <59762500+addcontent@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The connection-layer DNS-rebinding guard (_SSRFSafeResolver / _SSRFSafeAdapter, PR #24759) was
mounted only on SafeWebBaseLoader. Two user-reachable image fetches validate the URL then fetch
it through the shared get_session() pool with the default resolver, so a TTL-0 rebinding answer
that passed validate_url reaches an internal address at connect:
- get_image_base64_from_url (utils/files.py): user image_url on every chat completion.
- load_url_image (routers/images.py, POST /api/v1/images/edit): user-supplied image field.
Add get_ssrf_safe_session() (a one-off aiohttp session mounting _SSRFSafeResolver) and use it
for both fetches, so the connect-time IP is re-validated and a rebound loopback / RFC1918 /
metadata address is rejected. The shared pool is left untouched for the admin-configured
image-generation callers, which legitimately reach internal hosts.
Co-authored-by: dhyabi2 <32069256+dhyabi2@users.noreply.github.com>
Co-authored-by: geo-chen <2404584+geo-chen@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Qdrant backends implemented delete(ids=...) as a payload filter on
metadata.id, but points are stored with the item id as the Qdrant point id
(see _create_points), and not every point carries an id in its payload.
Memory points store only {created_at} in metadata (KB metadata embeddings
likewise), so deleting a single memory matched nothing and left an orphaned
vector that kept being injected into RAG context.
Delete by point id instead: PointIdsList for the standard backend, and a
tenant-scoped HasIdCondition for multitenancy (point ids are unique, so tenant
isolation is preserved). Filter-based deletion is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add support for Valkey vector database
Signed-off-by: Riley Des <riley.desserre@improving.com>
* feat: add CLIENT SETNAME to Valkey vector store connections
Set client_name on GlideClientConfiguration for both the main client
and batch client so connections are identifiable in CLIENT LIST,
monitoring dashboards, and CloudWatch metrics.
Signed-off-by: Riley Des <riley.desserre@improving.com>
---------
Signed-off-by: Riley Des <riley.desserre@improving.com>
Firecrawl /search returns either `{"data": [...]}` (flat list — v1, and
what frost19k reported on #23966) or `{"data": {"web": [...]}}` (v2,
current production). The parser only handled the dict shape:
data = response.get('data') or {}
results = data.get('web') or []
On a list-shape response, `data.get('web')` raised AttributeError,
caught by the function's outer try/except, and `search_firecrawl`
silently returned []. Web search worked against v2 endpoints but is one
upstream-format-change away from failing closed again. Accept either.
Open WebUI's collection ACL accepted any unknown name as a
legacy/ephemeral collection. In Milvus multi-tenancy mode that name
becomes the `resource_id` and is interpolated unescaped into a SQL-like
Milvus expression — `resource_id == '<name>'` — so a name like
x' or resource_id != '' or resource_id == 'x
turns the filter into a tautology and returns every tenant's chunks
from the shared collection.
All collection names Open WebUI generates are UUIDs, SHA-256 hex
digests, or fixed-prefix variants of those — they all fit
[A-Za-z0-9_-]. Add a strict format check in
filter_accessible_collections (utils.py) that drops any name outside
that set before any ACL or vector-store lookup, applied even on the
admin bypass path. _validate_collection_access then surfaces the dropped
name as a 403.
As defense in depth, MilvusClient now validates resource_id at every
expression-construction site and escapes single quotes / backslashes in
any other string interpolated into a filter (delete ids, metadata
filter values). Non-string filter values are typed-checked instead of
str()-formatted.
Co-authored-by: Claude <noreply@anthropic.com>
validate_url() resolves DNS to check IPs but discards the result; the
HTTP client resolves again independently. Between those two lookups an
attacker can swap the DNS record from a public IP to an internal one
(DNS rebinding).
Push the IP-is-global check into the actual connection layer so the
validated resolution is the one used for the TCP connect:
- aiohttp (_fetch): _SSRFSafeResolver wraps DefaultResolver and rejects
non-global IPs at resolve time (zero TOCTOU window).
- requests (_scrape): _SSRFSafeAdapter mounts custom urllib3 connection
classes whose _new_conn resolves, validates, and connects to the
validated IP in one shot (zero TOCTOU window).
Both paths respect ENABLE_RAG_LOCAL_WEB_FETCH (skip validation when on).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
SafePlaywrightURLLoader validated only the initially submitted URL and
then let the browser follow HTTP redirects and client-side navigations
without re-checking them, so a public URL could redirect into the
internal network (cloud metadata, RFC1918, loopback). Intercept
document-type requests, re-run validate_url on each, and apply the same
redirect policy as the requests loader (blocked unless
AIOHTTP_CLIENT_ALLOW_REDIRECTS). Sub-resource requests pass through
unchanged so page rendering performance is unaffected.
Co-authored-by: POV9en <POV9en@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
