perf: parse scraped web pages off the event loop (#27446)

`alazy_load()` builds every BeautifulSoup tree inline in an async function, so a web search that pulls in ten pages stops the entire worker for the whole time it spends parsing. Nothing else on that worker runs during it: not other users' token streams, not health checks, not socket.io traffic. Parsing is CPU work and it belongs in a thread.

Measured over 37 real pages, 13.5 MiB total, with a 5ms ticker sampling event-loop lag:

| | wall | worst loop stall | ticker fired |
|---|---|---|---|
| inline, html.parser (today) | 1793.8ms | 1788.8ms | 1 time |
| offloaded, html.parser | 1872.9ms | 82.9ms | 88 times |
| inline, lxml | 1346.7ms | 1341.8ms | 1 time |
| offloaded, lxml | 1445.4ms | 37.0ms | 118 times |

Today the loop is not merely slow during a batch, it is gone: a 5ms timer fired exactly once across 1.8 seconds. After the change it fires normally and the worst single stall drops by a factor of 20 to 36. The cost is 4 to 7 percent more wall time for the batch itself, from the thread handoffs, which is the right trade for a server handling more than one user.

Three details behind the shape of the change:

`get_text()` is only 2 percent of the cost (34ms against 1706ms of parsing over the corpus), so the whole per-page unit moves into the thread rather than the parse alone. Splitting them measured worse on both axes.

The offload is per page, not per batch. Handing the whole batch to one thread measured worse than either (2081ms wall, 235ms worst stall), so the loop is yielded to between pages.

The metadata block in `alazy_load()` was a duplicate of the module-level `extract_metadata()`, field for field, and `lazy_load()` was already using the shared helper. The new helper calls it too, which is why the diff removes more lines than it adds. The `ascrape_all()` override goes with it: it was a verbatim copy of the inherited implementation and `alazy_load()` was its only caller, so anything still calling it now gets the identical parent method, which resolves `self._unpack_fetch_results` to the override this class keeps.

Verified by feeding the real loader a 37 page corpus and comparing every resulting Document against the implementation this replaces:

```
PASS  one Document per url (37)
PASS  every Document identical to the pre-change implementation (0 differ)
PASS  parsing ran off the main thread
PASS  event loop kept running during parsing (90 ticks)
```

Both `page_content` and `metadata` are byte-identical on all 37 pages. This is independent of the parser in use and composes with switching the default parser to lxml: that change makes the stalls shorter, this one takes them off the loop.

backend/open_webui/retrieval/web/utils.py

@@ -800,11 +800,6 @@ class SafeWebBaseLoader(WebBaseLoader):
             final_results.append(BeautifulSoup(result, url_parser, **self.bs_kwargs))
         return final_results
 
-    async def ascrape_all(self, urls: List[str], parser: Union[str, None] = None) -> List[Any]:
-        """Async fetch all urls, then return soups for all results."""
-        results = await self.fetch_all(urls)
-        return self._unpack_fetch_results(results, urls, parser=parser)
-
     def lazy_load(self) -> Iterator[Document]:
         """Lazy load text from the url(s) in web_path with error handling."""
         for path in self.web_paths:
@@ -820,19 +815,20 @@ class SafeWebBaseLoader(WebBaseLoader):
                 # Log the error and continue with the next URL
                 log.exception(f'Error loading {path}: {e}')
 
+    def _document_from_html(self, html: str, url: str) -> Document:
+        """Build one Document."""
+        soup = self._unpack_fetch_results([html], [url])[0]
+        return Document(
+            page_content=soup.get_text(**self.bs_get_text_kwargs),
+            metadata=extract_metadata(soup, url),
+        )
+
     async def alazy_load(self) -> AsyncIterator[Document]:
         """Async lazy load text from the url(s) in web_path."""
-        results = await self.ascrape_all(self.web_paths)
-        for path, soup in zip(self.web_paths, results):
-            text = soup.get_text(**self.bs_get_text_kwargs)
-            metadata = {'source': path}
-            if title := soup.find('title'):
-                metadata['title'] = title.get_text()
-            if description := soup.find('meta', attrs={'name': 'description'}):
-                metadata['description'] = description.get('content', 'No description found.')
-            if html := soup.find('html'):
-                metadata['language'] = html.get('lang', 'No language found.')
-            yield Document(page_content=text, metadata=metadata)
+        results = await self.fetch_all(self.web_paths)
+        for path, html in zip(self.web_paths, results):
+            # parsing a large page costs hundreds of ms, keep it off the event loop
+            yield await asyncio.to_thread(self._document_from_html, html, path)
 
     async def aload(self) -> list[Document]:
         """Load data into Document objects."""