Mirrors/PowerToys
1
0
Fork 0
mirror of https://github.com/microsoft/PowerToys.git synced 2026-08-29 10:09:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

[UITests][PowerRename] Migrate to new .Next and add more UI tests (#50096)

Browse Source 
## Summary of the Pull Request

Adds a `PowerRename.UITests.Next` suite powered by winappcli and
automates all 18 scenarios from #40663. The suite covers PowerRename
settings, search and replace behavior, regular expressions, formatting
and filtering options, file-list interactions, and both classic and
Windows 11 context-menu workflows.

The PR also stabilizes shared `UITestAutomation.Next` runner lifetimes
and settings restoration, adds automation IDs for the original and
renamed counters, and prepares unsigned CI builds for PowerRename shell
testing. CI now signs the sparse context-menu MSIX and the
runner/Settings IPC companions with a disposable machine-trusted test
identity.

## PR Checklist

- [x] Closes: #40663
- [x] **Communication:** I've discussed this with core contributors
already. If the work hasn't been agreed, this work might be rejected
- [x] **Tests:** Added/updated and all pass
- [x] **Localization:** All end-user-facing strings can be localized
- [x] **Dev docs:** Added/updated
- [x] **New binaries:** Not applicable; no new shipped product binaries
are added
   - [x] JSON for signing: Not applicable
   - [x] WXS for installer: Not applicable
- [x] YML for CI pipeline: The new UI-test project is discovered through
the existing `*UITest*.csproj` pipeline flow and is registered in
`PowerToys.slnx`
   - [x] YML for signed pipeline: Not applicable
- [x] **Documentation updated:** Not applicable; there are no
user-facing behavior or documentation changes

## Detailed Description of the Pull Request / Additional comments

### PowerRename UI tests

- Adds a 25-case `PowerRename.UITests.Next` executable covering all 18
checklist items from #40663.
- Exercises classic context-menu registration on Windows 10 and Windows
11.
- Exercises the signed Windows 11 tier-1 context menu, including icon
visibility and real invocation with an Explorer selection.
- Covers search/replace preview and application, text formatting,
file/folder/subfolder inclusion, filename/extension scope, enumeration,
case sensitivity, match-all behavior, regular expressions, file
timestamps, Boost syntax, MRU autocomplete, persisted values, and
file-list selection/filtering.
- Preserves the existing legacy tests.

### Test reliability and automation hooks

- Reuses one runner/Settings lifetime across the complete PowerRename
suite to avoid repeated cold launches on constrained agents.
- Retains and restores global settings for the full class lifetime and
verifies that both Settings and the runner remain healthy.
- Propagates scope and PowerRename cleanup failures instead of silently
leaking process or profile state.
- Adds `OriginalCount` and `RenamedCount` automation IDs. These are
automation-only metadata and do not change the visible UI.
- Uses stable preview samples, exact count targeting, authoritative
Explorer selection, readable classic-menu inventories, and live UIA
visibility for popup items.

### Unsigned CI build support

- Extends the existing sparse-package test signer with required
Authenticode companion files.
- PowerRename jobs sign `PowerToys.exe` and `PowerToys.Settings.exe`
with the same disposable machine-trusted test identity used for sparse
MSIX packages.
- This preserves Release IPC authentication while allowing Settings
module-toggle commands to work on unsigned PR builds.
- Windows 11 and ARM64 PowerRename jobs require a validly signed
`PowerRenameContextMenuPackage.msix` before tests start.

## Validation Steps Performed

### Builds and discovery

- `PowerRename.UITests.Next` Debug x64 build: passed
- `PowerRename.UITests.Next` Debug ARM64 cross-build: passed
- `PowerRenameUI` Release x64 build: passed
- `UITestAutomation.Next.UnitTests` Debug x64 build: passed
- `UITestAutomation.Next.UnitTests`: 16/16 passed
- Microsoft.Testing.Platform discovery: 25 unique PowerRename test cases

### Local VM matrix

All runs used the complete unfiltered `TestCategory=PowerRename` suite
and restored the standard user's settings file byte-for-byte.

| Guest | Profile | Result |
|---|---|---:|
| Windows 10 x64 | Default, 4 vCPU / 8 GB | 25/25 | 
| Windows 10 x64 | Constrained, 1 vCPU / 4 GB | 25/25 | 
| Windows 11 x64 | Default, 4 vCPU / 8 GB | 25/25 | 
| Windows 11 x64 | Constrained, 1 vCPU / 4 GB | 25/25 | 

### Azure DevOps UI Test Automation

- Final build: [155646235 /
20260824.1](https://dev.azure.com/microsoft/Dart/_build/results?buildId=155646235)
- Source revision: `4496683104aea622b30179909bd6e95a17d5500f`
- ARM64: 25/25 passed
- Windows 10 x64: 25/25 passed
- Windows 11 x64: 25/25 passed
- Total: 75/75 passed, with zero failed, skipped, not-executed, or
unanalyzed results
- ARM64 and x64 Release product builds succeeded and published their
normal artifacts
- Signing steps verified the PowerRename sparse MSIX where applicable
and both IPC companion executables on every PowerRename test job

<img width="372" height="314" alt="image"
src="https://github.com/user-attachments/assets/bc303673-0325-4f85-8d70-ef93110baf5b"
/>
This commit is contained in:
Gleb Khmyznikov
2026-08-25 15:26:10 -07:00
committed by GitHub
parent 980d4193df
commit bea1b8e247
23 changed files with 3389 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
.github/skills/ui-tests-pipeline-ci/references/setup-preflight.md vendored Normal file
View File

@@ -0,0 +1,54 @@
# Azure DevOps setup preflight
Read this reference only when `Test-AzureDevOpsSetup.ps1` fails or the user asks what the readiness
check proves. The normal CI workflow needs only the invocation and pass criterion in
[agentic-loop.md](agentic-loop.md).
## Capability checks
| Check | What a pass proves |
|---|---|
| `PowerShell7` | The script is running on supported PowerShell 7+ semantics. |
| `AzureCLI` | `az` is installed, executable, and reports a parseable version. |
| `AzureDevOpsExtension` | The `azure-devops` CLI extension is installed, so artifact-download commands are available. |
| `CachedSignInAndToken` | The existing account can mint an Azure DevOps resource token without prompting. |
| `ProjectRead` | The identity can access organization `microsoft` and project `Dart`. |
| `PipelineDefinitionRead` | The enabled `UI Test Automation` definition is visible and uniquely resolved. |
| `BuildRead`, `TimelineRead`, `BuildLogsRead`, `ArtifactsRead` | Build diagnostics and pipeline artifacts are readable. |
| `TestRunsRead`, `TestResultsRead`, `TestAttachmentsRead` | Azure Test evidence endpoints are readable. An attachment count of zero is still a successful permission check. |
| `PipelinePreview` | The Run Pipeline API accepts the identity, branch, parameters, and template expansion. `id=-1` proves no build was created. |
| `RepeatedPromptFreeRead` | A second token-backed call completes without another authentication prompt. |
The preflight deliberately does **not** create a run, cancel a build, or retry/cancel a stage. Those
mutations consume resources or change tracked work and are verified only when the user's request
authorizes the real operation. After every authorized write, re-read the build/timeline and require
the expected state transition; a successful HTTP response alone is not proof that a delayed stage
retry materialized.
Do not use `az devops security permission show` or Azure DevOps Graph-user enumeration as the setup
gate. Resolving the current Graph descriptor can require the unrelated `ReadExtended Users`
permission, which many valid pipeline users do not have. A failure there does not mean pipeline
access is missing. The endpoint capability checks above are the authoritative, least-privilege
readiness proof.
## Failure remediation
The agent never starts an interactive sign-in or installs tools. If preflight fails, stop and report
the exact failed check. The user performs any required setup outside the agent, then the agent reruns
the same preflight.
| Failed check | Required remediation |
|---|---|
| `PowerShell7` | Install/use PowerShell 7 and invoke the script with `pwsh`, not Windows PowerShell. |
| `AzureCLI` | Install Azure CLI and open a new shell where `az version` succeeds. |
| `AzureDevOpsExtension` | User runs `az extension add --name azure-devops`, then reruns preflight. |
| `CachedSignInAndToken` | User signs into the Microsoft tenant with Azure CLI outside the agent. Never pass credentials through chat or run `az login` from the agent. |
| `ProjectRead` | Confirm the signed-in identity is a Microsoft FTE with access to `microsoft/Dart`. A valid token alone is insufficient. |
| `PipelineDefinitionRead` | Confirm project access and that `UI Test Automation` still exists and is enabled. |
| Build/log/artifact/test read | Request the missing Azure DevOps project/build/test permission; do not weaken evidence requirements. |
| `PipelinePreview` | Confirm the branch exists, the probe module is valid, and the identity can use/queue pipeline `UI Test Automation`. No run was created. |
No `az devops configure` defaults, PAT, `AZURE_DEVOPS_EXT_PAT`, service connection secret, or local
credential file is required. Every helper call supplies organization/project explicitly and obtains
the Azure DevOps token from the existing Azure CLI cache. Rerun preflight after switching accounts,
after token-cache changes, or immediately after any `401`/`403` response.