mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-09-01 19:51:34 +02:00
## Summary Adds an on-device **Phi Silica (Windows AI) paste provider** to Advanced Paste and richer per-action customization, plus the **package-identity plumbing** that lets the unpackaged Advanced Paste app use Windows AI APIs. > Note: this supersedes the earlier "self-contained MSIX package installed/registered by the > WiX installer" approach. Advanced Paste is **not** packaged or installed as a standalone > MSIX. It keeps shipping as the existing unpackaged, self-contained WinUI 3 executable in > `WinUI3Apps/` and acquires **package identity at runtime from the existing PowerToys sparse > package** — there are no installer or ESRP MSIX repack changes. ## Advanced Paste features - **New Phi Silica paste provider** (`CustomActions/PhiSilicaPasteProvider.cs`): an on-device AI provider backed by the Windows AI Phi Silica language model — no cloud endpoint or API key required. Registered as a new `AIServiceType` via `PasteAIProviderFactory` / `AIServiceTypeRegistry`. - **Additional custom actions** (`AdvancedPasteAdditionalAction`): user-defined actions with their own prompt, system prompt, AI provider, and shortcut — plus an optional "coaching" prompt/system-prompt/provider/shortcut and shortcut-conflict detection. - **Built-in default prompts** (`AdvancedPasteDefaultPrompts`) and updates to `AdvancedPasteCustomAction`, `PasteFormat(s)`, `OptionsViewModel`, and `PasteFormatExecutor` to support per-action provider selection and system prompts. - **Settings UI** (`AdvancedPastePage.xaml`/`.xaml.cs`, `Resources.resw`): configure the Phi Silica provider, choose a provider per action, edit system/coaching prompts, and a Phi Silica availability/readiness experience — Settings queries the Advanced Paste executable via `--check-phi-silica`, and a **"Download model"** action triggers `--prepare-phi-silica` to fetch the model and then re-probes. ## Package identity for Windows AI (replaces the MSIX-install approach) - Phi Silica is a **Limited Access Feature (LAF)** that can only be unlocked by a process with a registered **package identity**. Advanced Paste runs unpackaged, so it obtains identity from the existing **`Microsoft.PowerToys.SparseApp`** sparse package (`src/PackageIdentity/`): a new `<Application Id="PowerToys.AdvancedPasteUI">` entry in `AppxManifest.xml` maps it to `PowerToys.AdvancedPaste.exe`, with matching updates to `BuildSparsePackage.ps1`. - **LAF unlock** at runtime via `PhiSilicaLafHelper.cs`. The token/attestation are baked at build time by the `GeneratePhiSilicaLafCredentials` MSBuild target into `PhiSilicaLafCredentials.g.cs` — local **dev defaults** live in `src/PhiSilicaLaf.props` (imported from `Directory.Build.props`) and the **production secret** is injected via `/p:` in the release pipeline. - New **`AdvancedPaste.dev.manifest` / `AdvancedPaste.prod.manifest`** application manifests (selected by `CIBuild`) declaring full-trust and the system AI models capability. ## Build & pipeline - **Windows App SDK** moved to the coherent **stable `2.2.0`** line and **added `Microsoft.WindowsAppSDK.AI` `2.2.3`** (the Phi Silica APIs). Foundation `2.1.0` carries the sparse-identity PRI fix, and the stable AI build matches the OS Windows AI runtime. - **Independent versioning** for Advanced Paste (`src/modules/AdvancedPaste/custom.props`, XES one-store versioning, `AdvancedPasteVersion`). A `steps-setup-versioning.yml` step is added for Advanced Paste in `job-build-project.yml`, ordered **before** CmdPal to avoid a version-collision installer failure (WIX0103). - `release.yml` passes `PhiSilicaLafToken`/`PhiSilicaLafAttestation` into the main build; spell-check allow-list/patterns updated. - Removed now-unneeded dependencies: the `Microsoft.Windows.Compatibility` reference and the `Common.UI` "force matching DLL versions" hack. <img width="1073" height="716" alt="image" src="https://github.com/user-attachments/assets/9364d86a-c0d1-4a08-a669-d98cbcb4b140" /> <img width="1038" height="308" alt="image" src="https://github.com/user-attachments/assets/00eef1dd-b407-4841-bd64-f54f8a145c46" /> <img width="194" height="405" alt="image" src="https://github.com/user-attachments/assets/d2f9c5bb-2507-4112-b3e0-56da681f88ea" /> <img width="489" height="453" alt="image" src="https://github.com/user-attachments/assets/811d1afd-9993-48be-824e-82bb56c5ceca" /> [Video clip internal](https://onedrive.cloud.microsoft/✌️/a@9n6nl3fp/S/cQpvAHrL5M9ZR6eUawztyfyBEgUCwCF-aKg9TbKyyGWP4c0KMA) [Build internal](https://microsoft.visualstudio.com/Dart/_build/results?buildId=149920754&view=artifacts&pathAsName=false&type=publishedArtifacts) --------- Co-authored-by: Niels Laute <niels.laute@live.nl> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
101 lines
4.8 KiB
Markdown
101 lines
4.8 KiB
Markdown
# Advanced Paste – Phi Silica local testing
|
||
|
||
How to build, register, and test **Phi Silica** in **Advanced Paste (AP)** on a dev machine,
|
||
plus the few things that actually break it.
|
||
|
||
## How it fits together
|
||
|
||
AP ships as an **unpackaged, self-contained WinUI 3 exe** (`PowerToys.AdvancedPaste.exe`).
|
||
The Windows AI `LanguageModel` (Phi Silica) API is a **Limited Access Feature (LAF)**. For it
|
||
to work, all of these must line up:
|
||
|
||
1. **Package identity** — AP runs with identity granted by the sparse MSIX
|
||
`Microsoft.PowerToys.SparseApp`.
|
||
2. **Matching LAF creds** — the token/attestation baked into the exe match the registered
|
||
sparse package's publisher.
|
||
3. **AI metadata deployed** — the `Microsoft.Windows.AI*.winmd` files ship next to the exe;
|
||
the AI runtime resolves them **at runtime**.
|
||
4. **Model ready** — supported hardware and the on-device model downloaded
|
||
(`GetReadyState() == Ready`).
|
||
|
||
Two identities — the baked token must match the registered package's publisher:
|
||
|
||
| Build | Publisher Id | LAF creds |
|
||
|-------|--------------|-----------|
|
||
| **Dev** | `djwsxzxb4ksa8` | dev default in [`src/PhiSilicaLaf.props`](../../../src/PhiSilicaLaf.props) |
|
||
| **Prod** | `8wekyb3d8bbwe` | secret, injected only by `.pipelines/v2/release.yml` |
|
||
|
||
Non-secret pairing check: the exe's baked **Attestation** must equal the registered package's
|
||
**PublisherId**.
|
||
|
||
## Build + register (dev loop)
|
||
|
||
```powershell
|
||
$repo = "X:\GitHub\PowerToys"; $Plat = "ARM64"; $Cfg = "Debug" # or x64 / Release
|
||
|
||
# Build AP only (C#; reuses existing C++ outputs):
|
||
dotnet restore "$repo\src\modules\AdvancedPaste\AdvancedPaste\AdvancedPaste.csproj" /p:Platform=$Plat
|
||
& "$repo\tools\build\build.cmd" -Path "$repo\src\modules\AdvancedPaste\AdvancedPaste" `
|
||
-Platform $Plat -Configuration $Cfg /p:BuildProjectReferences=false
|
||
|
||
# Register the dev sparse package (creates + trusts a dev cert, grants identity):
|
||
pwsh -ExecutionPolicy Bypass -File "$repo\src\PackageIdentity\BuildSparsePackage.ps1" `
|
||
-Platform $Plat -Configuration $Cfg -DevRegister
|
||
# Expect: PublisherId djwsxzxb4ksa8, IsDevelopmentMode True
|
||
```
|
||
|
||
## Check the API
|
||
|
||
`PowerToys.AdvancedPaste.exe` is a **GUI-subsystem** app — run directly in a console it prints
|
||
nothing and returns no exit code. **Redirect** stdout/stderr and wait:
|
||
|
||
```powershell
|
||
$exe = "$repo\$Plat\$Cfg\WinUI3Apps\PowerToys.AdvancedPaste.exe"
|
||
$o = "$env:TEMP\ap.out"; $e = "$env:TEMP\ap.err"
|
||
$p = Start-Process $exe '--check-phi-silica' -Wait -PassThru -WindowStyle Hidden `
|
||
-RedirectStandardOutput $o -RedirectStandardError $e
|
||
"exit=$($p.ExitCode) stdout=$((Get-Content $o -Raw).Trim())"
|
||
Get-Content $e -Raw # stderr: [phi-silica] LAF unlock status: <…>; ReadyState: <…>
|
||
```
|
||
|
||
| `--check-phi-silica` | `--prepare-phi-silica` (downloads the model) |
|
||
|----------------------|----------------------------------------------|
|
||
| `0` Available · `1` NotReady · `2` NotSupported / unlock failed | `0` Ready · `1` Failed · `2` NotSupported |
|
||
|
||
`--check` only reads state; use `--prepare` to trigger the model download (`EnsureReadyAsync`).
|
||
On failure it prints the `HRESULT` to stderr.
|
||
|
||
Confirm the running AP has identity:
|
||
|
||
```powershell
|
||
$apPid = (Get-Process PowerToys.AdvancedPaste -EA SilentlyContinue | Select-Object -First 1).Id
|
||
if ($apPid) { & "$repo\src\PackageIdentity\Check-ProcessIdentity.ps1" -ProcessId $apPid }
|
||
# Expect a PFN ending in the publisher id that matches the baked attestation
|
||
```
|
||
|
||
## What actually breaks it
|
||
|
||
- **Missing `.winmd` (most important).** The Windows AI runtime resolves
|
||
`Microsoft.Windows.AI*.winmd` from the app folder at runtime. If they aren't deployed,
|
||
`GetReadyState()` returns `NotReady` and `EnsureReadyAsync()` fails with
|
||
`RO_E_METADATA_NAME_NOT_FOUND` (`0x8000000F`) — even though identity, token, and the AI DLLs
|
||
are all correct. The build emits these winmd into `WinUI3Apps\`; the **installer must harvest
|
||
them** (`*.winmd` is in the inclusion list of
|
||
[`generateAllFileComponents.ps1`](../../../installer/PowerToysSetupVNext/generateAllFileComponents.ps1)).
|
||
Classic symptom: "works from the build output but not from the installer" → check that the
|
||
installed `WinUI3Apps\` contains `Microsoft.Windows.AI*.winmd`.
|
||
- **Dev/prod mismatch.** A dev-cred exe running against a prod sparse package (or vice versa)
|
||
makes the LAF unlock silently return `Unavailable`. Keep the exe and the registered package
|
||
the same flavor, and verify with the attestation == publisherId check above.
|
||
- **Forgot to redirect.** `--check-phi-silica` in a console prints nothing — that's the
|
||
GUI-subsystem quirk, not a result.
|
||
|
||
## Cleanup
|
||
|
||
```powershell
|
||
pwsh -ExecutionPolicy Bypass -File "$repo\src\PackageIdentity\BuildSparsePackage.ps1" -Unregister
|
||
```
|
||
|
||
⚠️ This removes any `Microsoft.PowerToys.SparseApp` registration, **including a prod one** from
|
||
an installer — reinstall/repair PowerToys to restore it.
|