mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-09-01 19:51:34 +02:00
a87f5b07a9
## Summary Adds an on-device **Phi Silica (Windows AI) paste provider** to Advanced Paste and richer per-action customization, plus the **package-identity plumbing** that lets the unpackaged Advanced Paste app use Windows AI APIs. > Note: this supersedes the earlier "self-contained MSIX package installed/registered by the > WiX installer" approach. Advanced Paste is **not** packaged or installed as a standalone > MSIX. It keeps shipping as the existing unpackaged, self-contained WinUI 3 executable in > `WinUI3Apps/` and acquires **package identity at runtime from the existing PowerToys sparse > package** — there are no installer or ESRP MSIX repack changes. ## Advanced Paste features - **New Phi Silica paste provider** (`CustomActions/PhiSilicaPasteProvider.cs`): an on-device AI provider backed by the Windows AI Phi Silica language model — no cloud endpoint or API key required. Registered as a new `AIServiceType` via `PasteAIProviderFactory` / `AIServiceTypeRegistry`. - **Additional custom actions** (`AdvancedPasteAdditionalAction`): user-defined actions with their own prompt, system prompt, AI provider, and shortcut — plus an optional "coaching" prompt/system-prompt/provider/shortcut and shortcut-conflict detection. - **Built-in default prompts** (`AdvancedPasteDefaultPrompts`) and updates to `AdvancedPasteCustomAction`, `PasteFormat(s)`, `OptionsViewModel`, and `PasteFormatExecutor` to support per-action provider selection and system prompts. - **Settings UI** (`AdvancedPastePage.xaml`/`.xaml.cs`, `Resources.resw`): configure the Phi Silica provider, choose a provider per action, edit system/coaching prompts, and a Phi Silica availability/readiness experience — Settings queries the Advanced Paste executable via `--check-phi-silica`, and a **"Download model"** action triggers `--prepare-phi-silica` to fetch the model and then re-probes. ## Package identity for Windows AI (replaces the MSIX-install approach) - Phi Silica is a **Limited Access Feature (LAF)** that can only be unlocked by a process with a registered **package identity**. Advanced Paste runs unpackaged, so it obtains identity from the existing **`Microsoft.PowerToys.SparseApp`** sparse package (`src/PackageIdentity/`): a new `<Application Id="PowerToys.AdvancedPasteUI">` entry in `AppxManifest.xml` maps it to `PowerToys.AdvancedPaste.exe`, with matching updates to `BuildSparsePackage.ps1`. - **LAF unlock** at runtime via `PhiSilicaLafHelper.cs`. The token/attestation are baked at build time by the `GeneratePhiSilicaLafCredentials` MSBuild target into `PhiSilicaLafCredentials.g.cs` — local **dev defaults** live in `src/PhiSilicaLaf.props` (imported from `Directory.Build.props`) and the **production secret** is injected via `/p:` in the release pipeline. - New **`AdvancedPaste.dev.manifest` / `AdvancedPaste.prod.manifest`** application manifests (selected by `CIBuild`) declaring full-trust and the system AI models capability. ## Build & pipeline - **Windows App SDK** moved to the coherent **stable `2.2.0`** line and **added `Microsoft.WindowsAppSDK.AI` `2.2.3`** (the Phi Silica APIs). Foundation `2.1.0` carries the sparse-identity PRI fix, and the stable AI build matches the OS Windows AI runtime. - **Independent versioning** for Advanced Paste (`src/modules/AdvancedPaste/custom.props`, XES one-store versioning, `AdvancedPasteVersion`). A `steps-setup-versioning.yml` step is added for Advanced Paste in `job-build-project.yml`, ordered **before** CmdPal to avoid a version-collision installer failure (WIX0103). - `release.yml` passes `PhiSilicaLafToken`/`PhiSilicaLafAttestation` into the main build; spell-check allow-list/patterns updated. - Removed now-unneeded dependencies: the `Microsoft.Windows.Compatibility` reference and the `Common.UI` "force matching DLL versions" hack. <img width="1073" height="716" alt="image" src="https://github.com/user-attachments/assets/9364d86a-c0d1-4a08-a669-d98cbcb4b140" /> <img width="1038" height="308" alt="image" src="https://github.com/user-attachments/assets/00eef1dd-b407-4841-bd64-f54f8a145c46" /> <img width="194" height="405" alt="image" src="https://github.com/user-attachments/assets/d2f9c5bb-2507-4112-b3e0-56da681f88ea" /> <img width="489" height="453" alt="image" src="https://github.com/user-attachments/assets/811d1afd-9993-48be-824e-82bb56c5ceca" /> [Video clip internal](https://onedrive.cloud.microsoft/✌️/a@9n6nl3fp/S/cQpvAHrL5M9ZR6eUawztyfyBEgUCwCF-aKg9TbKyyGWP4c0KMA) [Build internal](https://microsoft.visualstudio.com/Dart/_build/results?buildId=149920754&view=artifacts&pathAsName=false&type=publishedArtifacts) --------- Co-authored-by: Niels Laute <niels.laute@live.nl> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
121 lines
7.3 KiB
Markdown
121 lines
7.3 KiB
Markdown
# Advanced Paste
|
|
|
|
[Public overview - Microsoft Learn](https://learn.microsoft.com/en-us/windows/powertoys/advanced-paste)
|
|
|
|
## Quick Links
|
|
|
|
[All Issues](https://github.com/microsoft/PowerToys/issues?q=is%3Aopen%20label%3A%22Product-Advanced%20Paste%22)<br>
|
|
[Bugs](https://github.com/microsoft/PowerToys/issues?q=is%3Aopen%20label%3A%22Product-Advanced%20Paste%22%20label%3AIssue-Bug)<br>
|
|
[Pull Requests](https://github.com/microsoft/PowerToys/pulls?q=is%3Apr+is%3Aopen++label%3A%22Product-Advanced+Paste%22)
|
|
|
|
## Overview
|
|
|
|
Advanced Paste is a PowerToys module that provides enhanced clipboard pasting with formatting options and additional functionality.
|
|
|
|
## Implementation Details
|
|
|
|
[Source code](/src/modules/AdvancedPaste)
|
|
|
|
TODO: Add implementation details
|
|
|
|
### Paste with AI Preview
|
|
|
|
The "Show preview" setting (`ShowCustomPreview`) controls whether AI-generated results are displayed in a preview window before pasting. **The preview feature does not consume additional AI credits**—the preview displays the same AI response that was already generated, cached locally from a single API call.
|
|
|
|
The implementation flow:
|
|
1. User initiates "Paste with AI" action
|
|
2. A single AI API call is made via `ExecutePasteFormatAsync`
|
|
3. The result is cached in `GeneratedResponses`
|
|
4. If preview is enabled, the cached result is displayed in the preview UI
|
|
5. User can paste the cached result without any additional API calls
|
|
|
|
See the `ExecutePasteFormatAsync(PasteFormat, PasteActionSource)` method in `OptionsViewModel.cs` for the implementation.
|
|
|
|
## Debugging
|
|
|
|
Advanced Paste is an unpackaged, self-contained WinUI 3 app (`PowerToys.AdvancedPaste.exe`). To call Windows AI APIs (Phi Silica / `Microsoft.Windows.AI.Text.LanguageModel`) it acquires **package identity** at runtime via a shared sparse MSIX package (`Microsoft.PowerToys.SparseApp`).
|
|
|
|
### Running and attaching the debugger
|
|
|
|
1. Set the **Runner** project (`src/runner`) as the startup project in Visual Studio.
|
|
2. Launch the Runner (F5). This starts the PowerToys tray icon and loads all module interfaces.
|
|
3. Open Settings (right-click tray icon → Settings) and enable the **Advanced Paste** module if it isn't already. The module launches `PowerToys.AdvancedPaste.exe` in the background immediately.
|
|
4. In Visual Studio, go to **Debug → Attach to Process** (`Ctrl+Alt+P`) and attach to `PowerToys.AdvancedPaste.exe` (select **Managed (.NET Core)** debugger).
|
|
|
|
Alternatively, use the VS Code launch configuration **"Run AdvancedPaste"** from [.vscode/launch.json](/.vscode/launch.json) to launch the exe directly — but note that without the Runner, IPC and hotkeys won't work.
|
|
|
|
### Sparse package identity (local development)
|
|
|
|
#### Why is this needed?
|
|
|
|
- The `LanguageModel` API requires a Limited Access Feature (LAF) unlock, which only succeeds when the calling process has a matching package identity.
|
|
- Advanced Paste is an unpackaged, self-contained WinUI 3 app. The sparse package grants it identity without converting it to a full MSIX.
|
|
- The csproj uses `<ProjectPriFileName>PowerToys.AdvancedPaste.pri</ProjectPriFileName>` (matching the convention of other WinUI3 apps like ImageResizer). This requires WindowsAppSDK Foundation >= 2.0.22 ([PR #6376](https://github.com/microsoft/WindowsAppSDK/pull/6376)) which fixes MRT PRI lookup under sparse identity so `Application.LoadComponent` resolves custom-named PRI files instead of hard-coding `resources.pri`.
|
|
|
|
#### One-step dev setup
|
|
|
|
```powershell
|
|
pwsh src/PackageIdentity/BuildSparsePackage.ps1 -Platform ARM64 -Configuration Debug -DevRegister
|
|
```
|
|
|
|
`-DevRegister`:
|
|
1. Generates a dev certificate under `src/PackageIdentity/.user/` (first run only).
|
|
2. Auto-imports that certificate into `CurrentUser\TrustedPeople` and `CurrentUser\Root` so the OS grants sparse identity to AP (without trust, `GetPackageFamilyName` returns `APPMODEL_ERROR_NO_PACKAGE` and LAF unlock silently fails).
|
|
3. Removes any prior registration.
|
|
4. Rewrites the publisher in a temp copy of `AppxManifest.xml` to match the dev cert subject.
|
|
5. Registers via `Add-AppxPackage -Register … -ExternalLocation X:\…\<Platform>\<Config>\WinUI3Apps`.
|
|
|
|
After registration verify:
|
|
|
|
```powershell
|
|
$pkg = Get-AppxPackage -Name '*SparseApp*'
|
|
$pkg.PackageFamilyName # Microsoft.PowerToys.SparseApp_<PublisherId>
|
|
$pkg.PublisherId # djwsxzxb4ksa8
|
|
$pkg.IsDevelopmentMode # True
|
|
```
|
|
|
|
Confirm AP picks up sparse identity at runtime:
|
|
|
|
```powershell
|
|
& 'ARM64\Debug\WinUI3Apps\PowerToys.AdvancedPaste.exe' --check-phi-silica
|
|
# Exit 0 = Available, 1 = NotReady, 2 = NotSupported
|
|
```
|
|
|
|
Re-register after rebuilding AP, changing `src/PackageIdentity/AppxManifest.xml`, or switching platforms/configurations by re-running the same command. Unregister with `-Unregister`.
|
|
|
|
#### Troubleshooting
|
|
|
|
| Problem | Cause | Fix |
|
|
|---------|-------|-----|
|
|
| `GetPackageFamilyName` returns `APPMODEL_ERROR_NO_PACKAGE` (15700) at runtime; LAF unlock returns `Unavailable` | Dev certificate not trusted (or sparse package not registered) | Re-run `BuildSparsePackage.ps1 -DevRegister` — auto-imports the cert into `TrustedPeople` and `Root`. |
|
|
| `Microsoft.UI.Xaml.dll` crash with `0xC000027B` (class-not-registered) on AP or Settings startup | `<Application>` `Executable` path in `src/PackageIdentity/AppxManifest.xml` does not resolve under the registered `ExternalLocation` (`<Config>\WinUI3Apps\`) | Confirm every `Executable` is relative to `WinUI3Apps\` (per #47177) and the file exists under the build output. |
|
|
| AP launches but never shows a window when triggered via hotkey | Runner's pipe-server wait timed out before AP's cold-start finished bootstrapping WinAppSDK + DI host | Already mitigated by the 15 s pipe timeout in `AdvancedPasteProcessManager.cpp`; warm-start launches connect in well under 1 s. |
|
|
| `XamlParseException` / `ms-appx:///Microsoft.UI.Xaml/Themes/…` not found | WindowsAppSDK Foundation < 2.0.22; MRT can't resolve custom PRI name under sparse identity | Ensure `Microsoft.WindowsAppSDK.Foundation` >= 2.0.22 in `Directory.Packages.props`. |
|
|
|
|
### How Settings UI checks Phi Silica availability
|
|
|
|
Settings UI does not have sparse package identity. To check whether Phi Silica is available, it launches Advanced Paste as a short-lived subprocess:
|
|
|
|
```
|
|
PowerToys.AdvancedPaste.exe --check-phi-silica
|
|
```
|
|
|
|
`Program.Main` recognizes this flag, calls `PhiSilicaLafHelper.TryUnlock()` + `LanguageModel.GetReadyState()`, prints one of `Available` / `NotReady` / `NotSupported` to stdout, and exits with the matching code (0/1/2). Settings reads stdout with a 10 s wait. Because each call is a fresh process, transient `Unavailable` results are not cached across checks.
|
|
|
|
### See also
|
|
|
|
- [Phi Silica local testing & troubleshooting guide](advancedpaste-phisilica-local-testing.md) — layer-by-layer diagnostics for Phi Silica availability
|
|
- [`src/PackageIdentity/readme.md`](/src/PackageIdentity/readme.md) — full sparse package documentation
|
|
- [microsoft/microsoft-ui-xaml#10856](https://github.com/microsoft/microsoft-ui-xaml/issues/10856) — original WinUI sparse-identity PRI bug
|
|
- [microsoft/WindowsAppSDK#6376](https://github.com/microsoft/WindowsAppSDK/pull/6376) — MRT sparse PRI fix (Foundation >= 2.0.22)
|
|
|
|
## Settings
|
|
|
|
| Setting | Description |
|
|
|---------|-------------|
|
|
| `ShowCustomPreview` | When enabled, shows AI-generated results in a preview window before pasting. Does not affect AI credit consumption. |
|
|
|
|
## Future Improvements
|
|
|
|
TODO: Add potential future improvements
|