mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-08-29 10:09:43 +02:00
[UITests] New framework around WinApp CLI, no WinAppDriver or Selenium. (#48467)
# Add winappcli-based UI test harness (no WinAppDriver / Selenium) ## Summary Introduces a new UI test harness — `Microsoft.PowerToys.UITest.Next` — that drives PowerToys modules through Microsoft's [winappcli](https://github.com/microsoft/WinAppCli) (UI Automation CLI) instead of WinAppDriver + Selenium. Engine is a single executable shelled out from C#; no third-party NuGet packages, no driver process, no Appium server. Adds two real consumers: a full ColorPicker end-to-end scenario and a Settings shell navigation smoke test. This is opt-in and additive — the existing `UITestAutomation` library and the WinAppDriver-based test projects are untouched. Both can coexist while we evaluate the new harness. Inspired in part by [#48414](https://github.com/microsoft/PowerToys/pull/48414), which lands the same architectural bet (winappcli, AutomationId selectors, no WinAppDriver) at a smaller scope. This PR generalizes it into a reusable library. ## Why WinAppDriver + Selenium is a legacy pre-agentic solution that is no longer actively maintained. It's unreliable, heavyweight, and slow. To achieve 100% UI test coverage, we should leverage modern, reliable solutions, and WinApp CLI is a strong candidate. ## What's in this PR ### Harness library — [`src/common/UITestAutomation.Next/`](src/common/UITestAutomation.Next/) | File | Purpose | |---|---| | [`WinappCli.cs`](src/common/UITestAutomation.Next/WinappCli.cs) | Process wrapper around `winapp.exe`. `Invoke` / `InvokeAssertSuccess` / `InvokeJson` / `IsAvailable` / `TryResolveExecutable`. `Result` carries the args and emits `DescribeFailure()` like `winapp ui invoke X -w 12345 -> exit 1; stderr: ...` | | [`Session.cs`](src/common/UITestAutomation.Next/Session.cs) | Test session, scoped by either HWND (`-w`) or process (`-a`) via `TargetScope`. `Find<T>` / `FindAll<T>` / `Inspect` / `Screenshot` / `SendKeys`. `Session.FromProcess(...)` factory for the single-window-per-process case | | [`SessionHelper.cs`](src/common/UITestAutomation.Next/SessionHelper.cs) | Owns the launch + window-readiness flow. Static `EnsureRunning(scope, timeout)` returns whether the call had to launch (so cleanup only kills what we started). Uses `UseShellExecute=true` so child handles don't keep MSTest hanging | | [`UITestBase.cs`](src/common/UITestAutomation.Next/UITestBase.cs) | MSTest base class. Pre-flights `WinappCli.IsAvailable()` once per process and fails fast with the install hint if `winapp.exe` isn't on PATH | | [`Element/*.cs`](src/common/UITestAutomation.Next/Element/) | `Element`, `Button`, `ToggleSwitch`, `TextBox`, `NavigationViewItem`, `Window`. `Click` / `MouseClick` / `Focus` / `GetProperty` / `GetValue` / `HelpText` / `WaitForProperty` / `WaitForGone` plus coords (`X`/`Y`/`Width`/`Height`) | | [`By.cs`](src/common/UITestAutomation.Next/By.cs) | `By.Name` / `By.AccessibilityId` / `By.Id` / `By.Slug` | | [`Windows.cs`](src/common/UITestAutomation.Next/Windows.cs) | `WindowsFinder.ListAll` / `ListByApp` / `WaitForWindowByApp` / `WaitForWindowByProcess`. Notes the winappcli bug where unfiltered `list-windows` drops untitled windows | | [`WindowControl.cs`](src/common/UITestAutomation.Next/WindowControl.cs) | Tolerant Win32 helpers — `TryCloseByApp` / `TryFocusByApp` / `SafeCloseAndFocus` / `TryKillProcess` — for `finally` blocks | | [`KeyboardHelper.cs`](src/common/UITestAutomation.Next/KeyboardHelper.cs) | Hybrid `keybd_event` + `SendKeys.SendWait` chord sender — required for global PowerToys hotkeys | | [`MouseHelper.cs`](src/common/UITestAutomation.Next/MouseHelper.cs) | `MoveTo` / `LeftClick` / `RightClick` / `LeftClickAt` Win32 wrappers | | [`ClipboardHelper.cs`](src/common/UITestAutomation.Next/ClipboardHelper.cs) | STA-thread `Clipboard` access with `WaitForText` | | [`ModuleConfigData.cs`](src/common/UITestAutomation.Next/ModuleConfigData.cs) | `PowerToysModule` enum + path/process-name resolution | ### Tests **[`src/modules/colorPicker/ColorPicker.UITests/`](src/modules/colorPicker/ColorPicker.UITests/)** — replaces the previous empty `UITest-ColorPicker` stub. One test, [`ColorPickerEndToEndTests.NavigateReadShortcutActivateAndCapture`](src/modules/colorPicker/ColorPicker.UITests/ColorPickerEndToEndTests.cs), drives the full E2E: 1. Navigate to the Color Picker page via the dashboard utilities stack 2. Toggle the module OFF, verify `PowerToys.ColorPickerUI` exits; toggle ON, verify it respawns 3. Read the activation shortcut from the page's `ShortcutControl` (`EditButton.HelpText`) 4. Clear clipboard, park cursor, send the chord 5. Wait for the picker overlay window 6. Read the displayed HEX from a hidden XAML automation peer (see below) 7. Left-click to capture; assert the clipboard value matches the peer's HEX 8. Wait for the editor window and assert the captured color appears in its tree **[`src/settings-ui/Settings.UITests/`](src/settings-ui/Settings.UITests/)** — `SettingsNavigationSmokeTests.NavigationItem_NavigatesWithoutCrashing` is one `[TestMethod]` parameterized with `[DynamicData]`, producing 31 discrete results — one per `NavigationViewItem` in [`ShellPage.xaml`](src/settings-ui/Settings.UI/SettingsXAML/Views/ShellPage.xaml). For each item: navigate, settle 250ms, assert `PowerToys.Settings` is still alive. Catches FailFast regressions in `ShellViewModel.Frame_NavigationFailed` that pure-logic unit tests can't reach (the failure path needs a `NavigationFailedEventArgs` which is a sealed WinRT projection). ### Product change **[`src/modules/colorPicker/ColorPickerUI/Views/MainView.xaml`](src/modules/colorPicker/ColorPickerUI/Views/MainView.xaml)** — adds a hidden `TextBlock` automation peer: ```xml <TextBlock x:Name="ColorHexAutomationPeer" AutomationProperties.AutomationId="ColorHexAutomationPeer" IsHitTestVisible="False" Opacity="0" Text="{Binding ColorText}" /> ``` The visible `ColorTextBlock` has `AutomationProperties.Name="{Binding ColorName}"`, which masks the HEX value in the UIA tree (you see "White" instead of `#FFFFFF`). This zero-impact peer mirrors `ColorText` so tests can read the actually-displayed HEX. `Opacity=0` + `IsHitTestVisible=False` keep it out of the visual layout and out of accessibility focus. ### Project wiring - [`PowerToys.slnx`](PowerToys.slnx) — registers `UITestAutomation.Next` under `/common/`, `ColorPicker.UITests` under `/modules/colorpicker/Tests/`, and `Settings.UITests` under `/settings-ui/Tests/`. Original `UITest-ColorPicker` stub csproj removed. - [`.github/actions/spell-check/expect.txt`](.github/actions/spell-check/expect.txt) — adds `winapp` / `winappcli`. ### Not in this PR - No pipeline changes. `winapp.exe` is expected to be pre-staged on the test agent image. If it's missing, `UITestBase` fails the first test with the install hint (`winget install Microsoft.winappcli`) rather than producing 30 opaque per-test errors. - No changes to the legacy `UITestAutomation` library or any of the existing `*.UITests` projects. ## Validation - All three projects build clean on `x64|Debug` (empty `build.<config>.<plat>.errors.log`): - `src/common/UITestAutomation.Next/` - `src/modules/colorPicker/ColorPicker.UITests/` - `src/settings-ui/Settings.UITests/` - Both tests run in Test Explorer / `dotnet test` via Microsoft.Testing.Platform (already enabled repo-wide in `Directory.Build.props`). - Local runs: ColorPicker E2E green; Settings smoke green across all 31 nav items. - `winapp 0.3.2` from `winget install Microsoft.winappcli`. ## Notes for reviewers - **`UseShellExecute = true`** in `SessionHelper.EnsureRunning` is intentional — `false` makes child processes inherit the test host's stdin/stdout/stderr handles, which keeps MTP/Test Explorer marking the run as "in progress" until the spawned PowerToys exits. - **Process-scope (`-a`) targeting** in the Settings smoke test handles single-instance handoff: the EXE you launch may exit with code 0 immediately after signalling an existing owner, so the alive check uses `Process.GetProcessesByName` rather than the launcher PID. - **AutomationId-only selectors** in the Settings smoke list keep the test localization-independent. Parent groups have `SelectsOnInvoked="False"` and only expand on click — `Element.Click` tries `InvokePattern → TogglePattern → SelectionItemPattern → ExpandCollapsePattern` so the same call works for both leaves and groups. - **Untitled-window discovery**: filtered `winapp ui list-windows -a <name>` returns windows that the unfiltered call drops (e.g. ColorPicker editor). `WindowsFinder.ListByApp` uses the filtered form. Reported upstream. ## Before Merge - Add the `winappcli` install step to the UI-test pipeline.
This commit is contained in:
Gleb Khmyznikov
71
.pipelines/InstallWinAppCli.ps1
Normal file
71
.pipelines/InstallWinAppCli.ps1
Normal file
@@ -0,0 +1,71 @@
|
||||
[CmdletBinding()]
|
||||
Param(
|
||||
# Target architecture: 'x64' or 'arm64'. Defaults to the pipeline's BuildPlatform variable.
|
||||
[string]$Platform = $env:BuildPlatform
|
||||
)
|
||||
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Pinned to the winappcli version the UITestAutomation.Next harness is validated against. Using
|
||||
# the standalone CLI zip (rather than the MSIX / winget) keeps this working on agents that lack
|
||||
# the App Installer and avoids MSIX registration entirely.
|
||||
$Version = 'v0.3.2'
|
||||
$NormalizedPlatform = if ([string]::IsNullOrWhiteSpace($Platform)) { 'x64' } else { $Platform.ToLowerInvariant() }
|
||||
|
||||
switch ($NormalizedPlatform)
|
||||
{
|
||||
'arm64'
|
||||
{
|
||||
$Asset = 'winappcli-arm64.zip'
|
||||
$ExpectedHash = 'dfe9d6eb70618665e4adcee989be8ecd076bfd387714a35a5b38597196fed093'
|
||||
}
|
||||
default
|
||||
{
|
||||
$Asset = 'winappcli-x64.zip'
|
||||
$ExpectedHash = '231373a4605ce7749172a70534ebab9305f91116e7f68d25cc73051372a6c579'
|
||||
}
|
||||
}
|
||||
|
||||
$DownloadUrl = "https://github.com/microsoft/winappCli/releases/download/$Version/$Asset"
|
||||
$ZipPath = Join-Path $env:Temp $Asset
|
||||
$InstallDir = Join-Path $env:Temp 'winappcli'
|
||||
|
||||
Write-Host "Downloading winappcli $Version ($Asset) from $DownloadUrl"
|
||||
Invoke-WebRequest -Uri $DownloadUrl -OutFile $ZipPath
|
||||
|
||||
# Verify the download against the published SHA256 before trusting it.
|
||||
$Hash = (Get-FileHash -Algorithm SHA256 $ZipPath).Hash
|
||||
if ($Hash -ne $ExpectedHash)
|
||||
{
|
||||
throw "$Asset has unexpected SHA256 hash: $Hash (expected $ExpectedHash)"
|
||||
}
|
||||
|
||||
# Fresh extract each run so a stale copy can't shadow the pinned version.
|
||||
if (Test-Path $InstallDir)
|
||||
{
|
||||
Remove-Item $InstallDir -Recurse -Force
|
||||
}
|
||||
Expand-Archive -Path $ZipPath -DestinationPath $InstallDir -Force
|
||||
|
||||
# Clear Mark-of-the-Web in case the agent applied it, so the CLI runs non-interactively.
|
||||
Get-ChildItem -Path $InstallDir -Recurse | Unblock-File -ErrorAction SilentlyContinue
|
||||
|
||||
$winapp = Get-ChildItem -Path $InstallDir -Recurse -Filter 'winapp.exe' | Select-Object -First 1 -ExpandProperty FullName
|
||||
if (-not $winapp)
|
||||
{
|
||||
throw "winapp.exe was not found after extracting $Asset to $InstallDir."
|
||||
}
|
||||
|
||||
Write-Host "winappcli installed at: $winapp"
|
||||
|
||||
# The harness (WinappCli.TryResolveExecutable) checks WINAPP_CLI_PATH first; also prepend the
|
||||
# folder to PATH so any other consumer in later steps resolves winapp.exe too.
|
||||
Write-Host "##vso[task.setvariable variable=WINAPP_CLI_PATH]$winapp"
|
||||
Write-Host "##vso[task.prependpath]$(Split-Path -Parent $winapp)"
|
||||
|
||||
& $winapp --version
|
||||
if ($LASTEXITCODE -ne 0)
|
||||
{
|
||||
throw "winapp.exe failed to run ('--version' exited with $LASTEXITCODE)."
|
||||
}
|
||||
@@ -171,6 +171,11 @@ jobs:
|
||||
fetchTags: false
|
||||
fetchDepth: 1
|
||||
|
||||
# Checkout to surface a missing import before full build.
|
||||
- pwsh: |-
|
||||
& '.pipelines/verifyCommonProps.ps1' -sourceDir '$(build.sourcesdirectory)\src'
|
||||
displayName: Audit shared common props for CSharp projects in src sub-folder
|
||||
|
||||
- ${{ if eq(parameters.enableMsBuildCaching, true) }}:
|
||||
- pwsh: |-
|
||||
$MSBuildCacheParameters = ""
|
||||
@@ -464,11 +469,6 @@ jobs:
|
||||
flattenFolders: True
|
||||
OverWrite: True
|
||||
|
||||
# Check if all projects (located in src sub-folder) import common props
|
||||
- pwsh: |-
|
||||
& '.pipelines/verifyCommonProps.ps1' -sourceDir '$(build.sourcesdirectory)\src'
|
||||
displayName: Audit shared common props for CSharp projects in src sub-folder
|
||||
|
||||
# Check if deps.json files don't reference different dll versions.
|
||||
- pwsh: |-
|
||||
& '.pipelines/verifyDepsJsonLibraryVersions.ps1' -targetDir '$(build.sourcesdirectory)\$(BuildPlatform)\$(BuildConfiguration)'
|
||||
|
||||
@@ -106,12 +106,19 @@ jobs:
|
||||
- template: steps-ensure-dotnet-version.yml
|
||||
parameters:
|
||||
sdk: true
|
||||
version: '9.0'
|
||||
version: '10.0'
|
||||
|
||||
- pwsh: |-
|
||||
& '$(build.sourcesdirectory)\.pipelines\InstallWinAppDriver.ps1'
|
||||
displayName: Download and install WinAppDriver
|
||||
|
||||
# winappcli (winapp.exe) powers the Microsoft.PowerToys.UITest.Next harness and isn't baked
|
||||
# into the agent image yet. winget / App Installer isn't available on these agents, so download
|
||||
# the pinned standalone CLI from its GitHub release. Drop this step once the CLI is pre-staged.
|
||||
- pwsh: |-
|
||||
& '$(build.sourcesdirectory)\.pipelines\InstallWinAppCli.ps1' -Platform '$(BuildPlatform)'
|
||||
displayName: Download and install winappcli (winapp.exe)
|
||||
|
||||
- ${{ if ne(parameters.buildSource, 'buildNow') }}:
|
||||
- task: DownloadPipelineArtifact@2
|
||||
inputs:
|
||||
@@ -149,7 +156,124 @@ jobs:
|
||||
inputs:
|
||||
displaySettings: 'optimal'
|
||||
|
||||
- script: |
|
||||
dotnet test $(Build.SourcesDirectory)\src\modules\fancyzones\FancyZones.UITests\FancyZones.UITests.csproj --no-build -c $(BuildConfiguration) -p:Platform=$(BuildPlatform)
|
||||
dotnet test $(Build.SourcesDirectory)\src\modules\fancyzones\FancyZonesEditor.UITests\FancyZonesEditor.UITests.csproj --no-build -c $(BuildConfiguration) -p:Platform=$(BuildPlatform)
|
||||
# Start WinAppDriver once for the whole job — WinAppDriver's documented CI pattern
|
||||
# (https://github.com/microsoft/WinAppDriver/blob/master/Docs/CI_AzureDevOps.md). Launching it
|
||||
# detached gives it its own console whose stdin blocks, so it stays alive for the run instead of
|
||||
# reading EOF and exiting the moment it starts listening (the failure mode when a test host launches
|
||||
# it as a child). The legacy UITest harness reuses an already-listening instance rather than
|
||||
# relaunching it per test, so this removes the per-assembly launch cost. The winappcli-based .Next
|
||||
# tests don't use WinAppDriver. Best-effort: if the pre-start fails, each assembly still launches its own.
|
||||
- pwsh: |
|
||||
$winapp = "C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe"
|
||||
if (Test-Path $winapp) {
|
||||
Start-Process -FilePath $winapp
|
||||
|
||||
$deadline = (Get-Date).AddSeconds(30)
|
||||
$ready = $false
|
||||
while (-not $ready -and (Get-Date) -lt $deadline) {
|
||||
try {
|
||||
$client = [System.Net.Sockets.TcpClient]::new()
|
||||
$client.Connect('127.0.0.1', 4723)
|
||||
$ready = $client.Connected
|
||||
$client.Close()
|
||||
} catch {
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
|
||||
if ($ready) {
|
||||
Write-Host 'WinAppDriver is listening on 127.0.0.1:4723.'
|
||||
} else {
|
||||
Write-Host "##vso[task.logissue type=warning]WinAppDriver did not start listening on :4723 within 30s; tests will launch it themselves."
|
||||
}
|
||||
} else {
|
||||
Write-Host "##vso[task.logissue type=warning]WinAppDriver not found at $winapp; tests will launch it themselves."
|
||||
}
|
||||
displayName: Start WinAppDriver (shared, persistent)
|
||||
|
||||
- pwsh: |
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$artifactRoot = "$(Pipeline.Workspace)\$(TestArtifactsName)"
|
||||
if (-not (Test-Path $artifactRoot)) {
|
||||
Write-Host "##vso[task.logissue type=error]UI test artifact not found: $artifactRoot"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# uiTestModules is a template parameter; flatten it to a delimited string for the script.
|
||||
$modulesRaw = '${{ join(';', parameters.uiTestModules) }}'
|
||||
$modules = @()
|
||||
if (-not [string]::IsNullOrWhiteSpace($modulesRaw)) {
|
||||
$modules = $modulesRaw -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ }
|
||||
}
|
||||
|
||||
# Each UI test project is a Microsoft.Testing.Platform app; its entry assembly is paired
|
||||
# with a *.runtimeconfig.json. Recurse under the staged 'tests' folders (tolerates TFM/RID subfolders).
|
||||
$entries = Get-ChildItem -Path $artifactRoot -Filter '*.runtimeconfig.json' -File -Recurse -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.Name -like '*UITests*' -and $_.FullName -match '\\tests\\' }
|
||||
if ($modules.Count -gt 0) {
|
||||
$entries = $entries | Where-Object { $n = $_.Name; ($modules | Where-Object { $n -like "*$_*" }).Count -gt 0 }
|
||||
}
|
||||
|
||||
# Run each test assembly once (a project reference can copy a runner into a sibling's output).
|
||||
$entries = $entries | Sort-Object FullName | Group-Object Name | ForEach-Object { $_.Group[0] }
|
||||
|
||||
if (-not $entries) {
|
||||
Write-Host "##vso[task.logissue type=error]No UI test runners matched (modules: '$modulesRaw') under $artifactRoot"
|
||||
exit 1
|
||||
}
|
||||
|
||||
$resultsDir = "$(Common.TestResultsDirectory)"
|
||||
New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null
|
||||
|
||||
$failed = 0
|
||||
foreach ($rc in ($entries | Sort-Object FullName -Unique)) {
|
||||
$base = $rc.Name -replace '\.runtimeconfig\.json$', ''
|
||||
$dir = $rc.DirectoryName
|
||||
$exe = Join-Path $dir "$base.exe"
|
||||
$dll = Join-Path $dir "$base.dll"
|
||||
Write-Host "##[group]Run UI tests: $base"
|
||||
Push-Location $dir
|
||||
try {
|
||||
if (Test-Path $exe) {
|
||||
& $exe --report-trx --results-directory $resultsDir
|
||||
} elseif (Test-Path $dll) {
|
||||
& dotnet $dll --report-trx --results-directory $resultsDir
|
||||
} else {
|
||||
Write-Warning "No runner (exe/dll) found for $base in $dir"
|
||||
}
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Warning "UI tests reported failures for $base (exit $LASTEXITCODE)"
|
||||
$failed++
|
||||
}
|
||||
} finally {
|
||||
Pop-Location
|
||||
Write-Host "##[endgroup]"
|
||||
}
|
||||
}
|
||||
|
||||
if ($failed -gt 0) {
|
||||
Write-Host "##vso[task.logissue type=error]$failed UI test project(s) reported failures."
|
||||
exit 1
|
||||
}
|
||||
displayName: "Run UI Tests"
|
||||
# Expose 'platform' as an environment variable so the harness's EnvironmentConfig.IsInPipeline
|
||||
# is true and it captures failure media (screenshots / recording / logs). The legacy VSTest task
|
||||
# set `env: { platform: $(TestPlatform) }`; the MTP migration to this pwsh step dropped it.
|
||||
env:
|
||||
platform: $(TestPlatform)
|
||||
|
||||
- task: PublishTestResults@2
|
||||
displayName: "Publish UI Test Results"
|
||||
condition: always()
|
||||
inputs:
|
||||
testResultsFormat: VSTest
|
||||
testResultsFiles: '$(Common.TestResultsDirectory)/**/*.trx'
|
||||
mergeTestResults: true
|
||||
failTaskOnFailedTests: false
|
||||
|
||||
# Stop the shared WinAppDriver (paired with the start step above) so it doesn't linger on the
|
||||
# self-hosted agent between jobs. Best-effort and always runs.
|
||||
- pwsh: |
|
||||
Get-Process -Name 'WinAppDriver' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
|
||||
displayName: Stop WinAppDriver
|
||||
condition: always()
|
||||
|
||||
Reference in New Issue
Block a user