Three searches LIKE against cast(json_col AS text), which means they have to match
bytes a JSON encoder wrote. Encoders disagree on non-ASCII: stdlib escapes it to
\uXXXX, orjson writes it raw. Which one produced a row depends on the codec in force
when it was written, so any single pattern finds only half the table.
models.py hard-codes the stdlib spelling, with a comment asserting SQLite stores
JSON via json.dumps(ensure_ascii=True). Model.meta is a JSONField, which has
serialised through JSONCodec since ENABLE_ORJSON was introduced, so on that setting
it stores raw UTF-8 and the escaped pattern matches nothing: non-ASCII workspace
model tag search is broken today. prompts.py and automations.py hard-code the
opposite spelling and miss rows written the other way.
json_text_variants returns both spellings a string can take inside serialised JSON,
collapsing to one for ASCII, and the three call sites OR over them. Rows written
under either setting are now found under either setting, which also covers a
database holding a mix of the two.
Case handling is unchanged. models.py keeps matching non-ASCII tags case-sensitively
on SQLite, whose LOWER() is ASCII-only and would not fold the stored text the way
str.lower() folds the tag. ASCII tags collapse to a single variant and take exactly
the query they took before.
Verified on SQLite across every combination of codec-that-wrote-the-row and
codec-the-app-is-running, for an ASCII and a CJK tag, over all three call sites: 24
of 24 match, against 12 of 24 before. Quoting still bounds whole-tag matches, so
searching "weather" does not match a row tagged "weathervane".
Co-authored-by: Claude <noreply@anthropic.com>
`get_message_list` moves through `messages_map` by key but tracked each message's own `id` field, which the message body does not have to carry. Track the key instead.
Raising GLOBAL_LOG_LEVEL to WARNING buys quieter output but not less work: 241 INFO call sites interpolate their payload into an f-string before the logging call gets to drop it. The heaviest is get_doc, which logs every chunk id and metadata dict in a collection, so on the full-context retrieval path that is the entire knowledge base, once per chat request.
That one line at WARNING, CPython 3.12:
| knowledge base | payload | before | after |
| -------------- | ------- | -------- | ------- |
| top-k of 3 | 1.2 kB | 3.8 us | 0.07 us |
| 500 chunks | 201 kB | 583.6 us | 0.08 us |
| 5000 chunks | 2.0 MB | 5.8 ms | 0.15 us |
The lazy form log.info('query_doc:result %s %s', result.ids, result.metadatas) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. Output at INFO is byte-identical. Two sites that already built their message eagerly, one str concat and one % operator, move to the same lazy form.
Docker compose list-form environment syntax passes quotes through verbatim, so WEB_FETCH_FILTER_LIST="" reaches the backend as two literal quote characters rather than an empty string. Config parsing turned that into the filter entry '""', which has no "!" prefix and therefore landed in the allow list. A non-empty allow list requires every host to match one of its entries, and a quotes-only pattern can never match a hostname, so every fetch_url and web loader request was rejected with "URL blocked by filter list" and surfaced to the user as "The URL you provided is invalid".
get_allow_block_lists now strips surrounding quote characters from each entry and drops entries that are empty after normalisation. Quoted but otherwise valid entries such as "example.com" or !"example.com" now behave as their unquoted forms, and garbage entries no longer convert the default blocklist into a match-nothing allowlist that blocks everything.
Fixes#26908
is_string_allowed does endswith() matching and was called with the full URL
(retrieval/web/utils.py) against WEB_FETCH_FILTER_LIST, so a blocklisted host with any
path (https://blocked.example/x) ended with /x, not the host, and slipped through; the
allowlist direction false-rejected legitimate URLs and admitted attacker URLs ending in
an allowed string. The same endswith caused label confusion at the hostname call site
(retrieval/web/main.py): corp.com matched evilcorp.com, 10.0.0.1 matched 110.0.0.1.
Add is_host_allowed(host, ...) matching on DNS label boundaries (host == pattern or
host.endswith('.' + pattern)), called with the parsed hostname at both web-fetch call
sites. is_string_allowed is left unchanged for the unrelated function-name filters
(utils/middleware.py, utils/tools.py).
The separate is_global guard (validate_url / _ssrf_safe_new_conn, active when
ENABLE_RAG_LOCAL_WEB_FETCH is off) already blocks RFC1918/loopback/link-local, so this
restores the admin's intended blocking of specific public hosts.
Co-authored-by: addcontent <59762500+addcontent@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After migration to async db operations, the throttle decorator also
needs to support async. Since the decorator is only used for async funcs
now, we can just switch it to async instead of supporting sync and async
at the same time.
Signed-off-by: Adam Tao <tcx4c70@gmail.com>
