mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-08-29 10:09:43 +02:00
[UITests] Add UITest.Next suites (Image Resizer, Peek, File Explorer add‑ons, File Locksmith) + local‑VM tooling and CI test‑signing (#49671)
## Summary Adds end‑to‑end UI tests on the `Microsoft.PowerToys.UITest.Next` (winappcli) framework for three modules, grows the shared `.Next` test framework with the helpers those suites needed, and adds the CI plumbing that lets shell‑extension tests exercise the **real** Windows 11 modern context menu. Also ships two agent skills that document how to write and run these tests. Product runtime behavior is **unchanged** — the only product edits are test‑observability hooks in Peek and a unit‑test project exclude. Closes: https://github.com/microsoft/PowerToys/issues/40660 https://github.com/microsoft/PowerToys/issues/49424 https://github.com/microsoft/PowerToys/issues/40661 ## What's added ### New UI test suites - **Image Resizer** — `src/modules/imageresizer/tests/ImageResizer.UITests`: context‑menu enable/disable tracking, the resize dialog, custom presets, every fit mode, every unit, filename format, keep‑date, shrink‑only, replace‑in‑place, and orientation. - **Peek** — `src/modules/peek/Peek.UITests.Next`: file‑preview coverage across image/text/archive/ markdown types with per‑arch visual baselines. - **File Explorer add‑ons** — `src/modules/previewpane/PreviewPane.UITests`: Preview Pane handlers and thumbnail providers. ### `UITestAutomation.Next` framework - New helpers: `ExplorerShell` (Shell selection/view‑mode interop), `WaitHelper` (structured stable waits), `WindowControl` (foreground/context‑menu/process control), `VisualAssert` (image compare), `WindowHelper`. - Updates to `Session`, `UITestBase`, `SettingsConfigHelper`, `WinappCli`. - New `UITestAutomation.Next.UnitTests` project covering the new wait/settings/CLI helpers. ### CI — sign sparse MSIX so the modern menu registers - **`.pipelines/signSparsePackages.ps1`** — self‑signs each sparse context‑menu MSIX with a publisher‑matching test certificate and force‑trusts it (machine stores), so `AddPackageByUriAsync` succeeds on otherwise‑unsigned PR builds. Robust `signtool` discovery with a NuGet fallback; test‑only trust that asserts no security. - Wired into **`.pipelines/v2/templates/job-test-project.yml`** as a best‑effort step covering the run‑in‑place, machine‑install, and per‑user‑install locations. Signs nothing it can't (skips already‑signed packages) and never fails the job. ### Product changes (test observability only) - **Peek `FilePreview.xaml` / `.xaml.cs`** — a named `LoadingIndicator` and a hidden automation peer that exposes the current preview state as text, so tests can read load state deterministically. No runtime behavior change. - **`ImageResizer.UnitTests.csproj`** — exclude the sibling `ImageResizer.UITests\**` folder from the unit‑test compilation. ### Agent skills & docs - **New `ui-tests-local-vm` skill** — run `.Next` suites in persistent dockur/windows VMs: setup, agentic loop, image customization, troubleshooting, the shell‑extension **signing** reference, plus controller/guest scripts and VM templates. - **Updated `ui-tests-migration` skill** — WinAppDriver/Selenium → `.Next` porting guidance (CI stability, Explorer/shell‑extension test design, patterns & pitfalls). - **`doc/devdocs/development/ui-tests.md`** — updated for the `.Next` workflow. ## Testing - All three suites pass locally and in CI across **x64 Win10**, **x64 Win11**, and **arm64** (machine and per‑user install legs). ## Reviewer notes - No product runtime behavior changes; product edits are limited to the Peek test hooks above. - The CI signing step is a **test‑only** trust anchor (self‑signed, scoped to the agent) and is best‑effort, so it can only add modern‑menu coverage and never regress the job.
This commit is contained in:
Gleb Khmyznikov
@@ -1,8 +1,38 @@
|
||||
# UI tests framework
|
||||
|
||||
A specialized UI test framework for PowerToys that makes it easy to write UI tests for PowerToys modules or settings. Let's start writing UI tests!
|
||||
PowerToys provides UI-test frameworks for modules and Settings. New tests should use
|
||||
`Microsoft.PowerToys.UITest.Next`, which drives Windows UI Automation through `winappcli` and runs as
|
||||
a Microsoft.Testing.Platform executable. The legacy `Microsoft.PowerToys.UITest` framework uses
|
||||
WinAppDriver/Selenium and remains documented for existing suites and migration baselines.
|
||||
|
||||
## Before running tests
|
||||
## Agent-assisted workflows
|
||||
|
||||
Two repository skills cover the complete implementation and validation loop:
|
||||
|
||||
- [UI-tests migration skill](../../../.github/skills/ui-tests-migration/SKILL.md): create new
|
||||
`.Next` test projects, port legacy WinAppDriver tests, design stable selectors/waits/lifecycle, and
|
||||
prepare tests for CI.
|
||||
- [Local-VM UI-tests skill](../../../.github/skills/ui-tests-local-vm/SKILL.md): create persistent
|
||||
Windows 10 and Windows 11 Hyper-V guests, stage current build/test artifacts, execute tests in a
|
||||
standard-user interactive desktop, and collect durable TRX/log/screenshot/video evidence.
|
||||
|
||||
For new or migrated tests, use both skills. Build first, then use the local VMs as the default live
|
||||
agentic loop: run one deterministic test, diagnose and fix it, and finally widen to the complete
|
||||
module suite on both supported Windows versions.
|
||||
|
||||
## Before running tests
|
||||
|
||||
### `.Next` tests
|
||||
|
||||
- Build the PowerToys runtime and `.UITests.Next` test executable.
|
||||
- Install the pinned `winappcli` runtime or set `WINAPP_CLI_PATH`. The pipeline helper is
|
||||
`.pipelines/InstallWinAppCli.ps1`.
|
||||
- Use a live interactive desktop. UIA, foreground input, Explorer, hotkeys, and rendering do not work
|
||||
in session 0.
|
||||
- Exit an existing PowerToys instance before a host-desktop run. The harness owns the runner and
|
||||
module lifecycle.
|
||||
|
||||
### Legacy tests
|
||||
|
||||
- Install Windows Application Driver v1.2.1 from https://github.com/microsoft/WinAppDriver/releases/tag/v1.2.1 to the default directory (`C:\Program Files (x86)\Windows Application Driver`)
|
||||
|
||||
@@ -10,12 +40,101 @@
|
||||
|
||||
## Running tests
|
||||
|
||||
### `.Next` tests
|
||||
|
||||
Build the focused project with the repository script, then run the produced Microsoft.Testing.Platform
|
||||
executable directly:
|
||||
|
||||
```pwsh
|
||||
tools\build\build.cmd `
|
||||
-Path src\modules\<Module>\Tests\<Module>.UITests.Next `
|
||||
-Platform x64 `
|
||||
-Configuration Debug
|
||||
|
||||
$exe = 'x64\Debug\tests\<Module>.UITests.Next\net10.0-windows10.0.26100.0\<Module>.UITests.Next.exe'
|
||||
& $exe `
|
||||
--filter 'TestCategory=<Module>' `
|
||||
--report-trx `
|
||||
--report-trx-filename module.trx `
|
||||
--results-directory .\TestResults\<Module> `
|
||||
--timeout 7m
|
||||
```
|
||||
|
||||
Use explicit filter properties such as `Name=`, `Name~`, `FullyQualifiedName~`, or `TestCategory=`.
|
||||
A bare display name can select zero tests. The `7m` timeout above is a focused-filter example; choose
|
||||
a larger value for a module or project-wide run.
|
||||
|
||||
### Legacy tests
|
||||
|
||||
- Exit PowerToys if it's running.
|
||||
|
||||
- Open `PowerToys.slnx` in Visual Studio and build the solution.
|
||||
|
||||
- Run tests in the Test Explorer (`Test > Test Explorer` or `Ctrl+E, T`).
|
||||
|
||||
## Running `.Next` tests in persistent local VMs
|
||||
|
||||
The supported local backend is a pair of persistent Hyper-V guests driven through PowerShell Direct:
|
||||
Windows 10 and Windows 11, each with an already logged-on standard-user desktop. The VMs reveal
|
||||
first-run, profile, Explorer, WebView2, foreground, and lifecycle assumptions without modifying the
|
||||
host profile, while retaining staged payloads for a fast edit/build/rerun loop.
|
||||
|
||||
### One-time host setup
|
||||
|
||||
Scaffold a VM root outside the repository, then follow the generated next steps to create the
|
||||
untracked configuration:
|
||||
|
||||
```pwsh
|
||||
pwsh .github\skills\ui-tests-local-vm\scripts\Initialize-LocalVm.ps1 `
|
||||
-DestinationRoot C:\PowerToysUiTestVm
|
||||
|
||||
pwsh .github\skills\ui-tests-local-vm\scripts\Initialize-LocalVmHost.ps1 `
|
||||
-VmRoot C:\PowerToysUiTestVm `
|
||||
-CheckOnly
|
||||
```
|
||||
|
||||
If `-CheckOnly` reports `IsReady=false`, a human must run the elevated setup command it prints.
|
||||
Hyper-V group membership, the DPAPI-protected guest administrator credential, and guest creation
|
||||
cannot be completed by an agent. See the [setup reference](../../../.github/skills/ui-tests-local-vm/references/setup.md)
|
||||
for install media, `vm.config.psd1`, Windows 10/11 guest creation, and baseline checkpoints.
|
||||
|
||||
### Run the agentic loop
|
||||
|
||||
Create a module exchange containing `ui-tests.zip`, `powertoys-runtime.zip`, `winappcli.zip`, and
|
||||
`dotnet-runtime.zip` as described in the
|
||||
[agentic-loop reference](../../../.github/skills/ui-tests-local-vm/references/agentic-loop.md). Payloads
|
||||
are extracted to guest-local storage; tests are never run directly from a host share.
|
||||
|
||||
```pwsh
|
||||
$vmRoot = 'C:\PowerToysUiTestVm'
|
||||
$exchange = "$vmRoot\shared\PowerToysUiTests\<Module>"
|
||||
|
||||
pwsh .github\skills\ui-tests-local-vm\scripts\Invoke-LocalVmUiTest.ps1 `
|
||||
-VmName PowerToysUiTest-Win11 `
|
||||
-ConfigurationPath "$vmRoot\vm.config.psd1" `
|
||||
-VmRoot $vmRoot `
|
||||
-ExchangeRoot $exchange `
|
||||
-TestExecutable '<Module>.UITests.Next.exe' `
|
||||
-Filter 'Name=<Module>.FocusedTest' `
|
||||
-Platform x64Win11 `
|
||||
-BuildLabel (git rev-parse HEAD) `
|
||||
-SuiteTimeout 15m `
|
||||
-TimeoutMinutes 25 `
|
||||
-ReuseStagedPayload
|
||||
```
|
||||
|
||||
The controller starts the guest if needed, validates the standard-user token, Explorer session, and
|
||||
desktop size, then runs the test through a limited interactive scheduled task. It streams progress
|
||||
and returns `status.json`, TRX counters, per-test failures, logs, screenshots, and retained failure
|
||||
recordings under `<ExchangeRoot>\LocalVmResults\<runId>`.
|
||||
|
||||
After each source change, rebuild and replace only the changed archive, then rerun the same focused
|
||||
filter with `-ReuseStagedPayload`. Widen only after that behavior is understood. A module is complete
|
||||
only after the full category filter passes with `executed == total` on both Windows 10 and Windows 11;
|
||||
restore the baseline checkpoint for the final clean-profile confirmation. See the local-VM
|
||||
[troubleshooting guide](../../../.github/skills/ui-tests-local-vm/references/troubleshooting.md) for
|
||||
desktop, PowerShell Direct, payload, shell-extension signing, and evidence failures.
|
||||
|
||||
## Running tests in pipeline
|
||||
|
||||
The PowerToys UI test pipeline provides flexible options for building and testing:
|
||||
@@ -68,6 +187,14 @@ The PowerToys UI test pipeline provides flexible options for building and testin
|
||||
- Pipeline: https://microsoft.visualstudio.com/Dart/_build?definitionId=161438&_a=summary
|
||||
|
||||
## How to add the first UI tests for your modules
|
||||
|
||||
Use the [UI-tests migration skill](../../../.github/skills/ui-tests-migration/SKILL.md) for new
|
||||
`.Next` projects and ports. It contains the current executable project scaffold, API mapping, naming,
|
||||
CI-stability checklist, and validated examples.
|
||||
|
||||
The project sample below describes the **legacy WinAppDriver framework** and is retained for existing
|
||||
legacy suites. Do not use it as the starting point for a new `.Next` project.
|
||||
|
||||
- Follow the naming convention: 
|
||||
- Create a new project and add the following references to the project file. Change the OutputPath to your own module's path.
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user