mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-08-29 10:09:43 +02:00
bea1b8e247
## Summary of the Pull Request Adds a `PowerRename.UITests.Next` suite powered by winappcli and automates all 18 scenarios from #40663. The suite covers PowerRename settings, search and replace behavior, regular expressions, formatting and filtering options, file-list interactions, and both classic and Windows 11 context-menu workflows. The PR also stabilizes shared `UITestAutomation.Next` runner lifetimes and settings restoration, adds automation IDs for the original and renamed counters, and prepares unsigned CI builds for PowerRename shell testing. CI now signs the sparse context-menu MSIX and the runner/Settings IPC companions with a disposable machine-trusted test identity. ## PR Checklist - [x] Closes: #40663 - [x] **Communication:** I've discussed this with core contributors already. If the work hasn't been agreed, this work might be rejected - [x] **Tests:** Added/updated and all pass - [x] **Localization:** All end-user-facing strings can be localized - [x] **Dev docs:** Added/updated - [x] **New binaries:** Not applicable; no new shipped product binaries are added - [x] JSON for signing: Not applicable - [x] WXS for installer: Not applicable - [x] YML for CI pipeline: The new UI-test project is discovered through the existing `*UITest*.csproj` pipeline flow and is registered in `PowerToys.slnx` - [x] YML for signed pipeline: Not applicable - [x] **Documentation updated:** Not applicable; there are no user-facing behavior or documentation changes ## Detailed Description of the Pull Request / Additional comments ### PowerRename UI tests - Adds a 25-case `PowerRename.UITests.Next` executable covering all 18 checklist items from #40663. - Exercises classic context-menu registration on Windows 10 and Windows 11. - Exercises the signed Windows 11 tier-1 context menu, including icon visibility and real invocation with an Explorer selection. - Covers search/replace preview and application, text formatting, file/folder/subfolder inclusion, filename/extension scope, enumeration, case sensitivity, match-all behavior, regular expressions, file timestamps, Boost syntax, MRU autocomplete, persisted values, and file-list selection/filtering. - Preserves the existing legacy tests. ### Test reliability and automation hooks - Reuses one runner/Settings lifetime across the complete PowerRename suite to avoid repeated cold launches on constrained agents. - Retains and restores global settings for the full class lifetime and verifies that both Settings and the runner remain healthy. - Propagates scope and PowerRename cleanup failures instead of silently leaking process or profile state. - Adds `OriginalCount` and `RenamedCount` automation IDs. These are automation-only metadata and do not change the visible UI. - Uses stable preview samples, exact count targeting, authoritative Explorer selection, readable classic-menu inventories, and live UIA visibility for popup items. ### Unsigned CI build support - Extends the existing sparse-package test signer with required Authenticode companion files. - PowerRename jobs sign `PowerToys.exe` and `PowerToys.Settings.exe` with the same disposable machine-trusted test identity used for sparse MSIX packages. - This preserves Release IPC authentication while allowing Settings module-toggle commands to work on unsigned PR builds. - Windows 11 and ARM64 PowerRename jobs require a validly signed `PowerRenameContextMenuPackage.msix` before tests start. ## Validation Steps Performed ### Builds and discovery - `PowerRename.UITests.Next` Debug x64 build: passed - `PowerRename.UITests.Next` Debug ARM64 cross-build: passed - `PowerRenameUI` Release x64 build: passed - `UITestAutomation.Next.UnitTests` Debug x64 build: passed - `UITestAutomation.Next.UnitTests`: 16/16 passed - Microsoft.Testing.Platform discovery: 25 unique PowerRename test cases ### Local VM matrix All runs used the complete unfiltered `TestCategory=PowerRename` suite and restored the standard user's settings file byte-for-byte. | Guest | Profile | Result | |---|---|---:| | Windows 10 x64 | Default, 4 vCPU / 8 GB | 25/25 | | Windows 10 x64 | Constrained, 1 vCPU / 4 GB | 25/25 | | Windows 11 x64 | Default, 4 vCPU / 8 GB | 25/25 | | Windows 11 x64 | Constrained, 1 vCPU / 4 GB | 25/25 | ### Azure DevOps UI Test Automation - Final build: [155646235 / 20260824.1](https://dev.azure.com/microsoft/Dart/_build/results?buildId=155646235) - Source revision: `4496683104aea622b30179909bd6e95a17d5500f` - ARM64: 25/25 passed - Windows 10 x64: 25/25 passed - Windows 11 x64: 25/25 passed - Total: 75/75 passed, with zero failed, skipped, not-executed, or unanalyzed results - ARM64 and x64 Release product builds succeeded and published their normal artifacts - Signing steps verified the PowerRename sparse MSIX where applicable and both IPC companion executables on every PowerRename test job <img width="372" height="314" alt="image" src="https://github.com/user-attachments/assets/bc303673-0325-4f85-8d70-ef93110baf5b" />
3.9 KiB
3.9 KiB
Azure DevOps setup preflight
Read this reference only when Test-AzureDevOpsSetup.ps1 fails or the user asks what the readiness
check proves. The normal CI workflow needs only the invocation and pass criterion in
agentic-loop.md.
Capability checks
| Check | What a pass proves |
|---|---|
PowerShell7 |
The script is running on supported PowerShell 7+ semantics. |
AzureCLI |
az is installed, executable, and reports a parseable version. |
AzureDevOpsExtension |
The azure-devops CLI extension is installed, so artifact-download commands are available. |
CachedSignInAndToken |
The existing account can mint an Azure DevOps resource token without prompting. |
ProjectRead |
The identity can access organization microsoft and project Dart. |
PipelineDefinitionRead |
The enabled UI Test Automation definition is visible and uniquely resolved. |
BuildRead, TimelineRead, BuildLogsRead, ArtifactsRead |
Build diagnostics and pipeline artifacts are readable. |
TestRunsRead, TestResultsRead, TestAttachmentsRead |
Azure Test evidence endpoints are readable. An attachment count of zero is still a successful permission check. |
PipelinePreview |
The Run Pipeline API accepts the identity, branch, parameters, and template expansion. id=-1 proves no build was created. |
RepeatedPromptFreeRead |
A second token-backed call completes without another authentication prompt. |
The preflight deliberately does not create a run, cancel a build, or retry/cancel a stage. Those mutations consume resources or change tracked work and are verified only when the user's request authorizes the real operation. After every authorized write, re-read the build/timeline and require the expected state transition; a successful HTTP response alone is not proof that a delayed stage retry materialized.
Do not use az devops security permission show or Azure DevOps Graph-user enumeration as the setup
gate. Resolving the current Graph descriptor can require the unrelated ReadExtended Users
permission, which many valid pipeline users do not have. A failure there does not mean pipeline
access is missing. The endpoint capability checks above are the authoritative, least-privilege
readiness proof.
Failure remediation
The agent never starts an interactive sign-in or installs tools. If preflight fails, stop and report the exact failed check. The user performs any required setup outside the agent, then the agent reruns the same preflight.
| Failed check | Required remediation |
|---|---|
PowerShell7 |
Install/use PowerShell 7 and invoke the script with pwsh, not Windows PowerShell. |
AzureCLI |
Install Azure CLI and open a new shell where az version succeeds. |
AzureDevOpsExtension |
User runs az extension add --name azure-devops, then reruns preflight. |
CachedSignInAndToken |
User signs into the Microsoft tenant with Azure CLI outside the agent. Never pass credentials through chat or run az login from the agent. |
ProjectRead |
Confirm the signed-in identity is a Microsoft FTE with access to microsoft/Dart. A valid token alone is insufficient. |
PipelineDefinitionRead |
Confirm project access and that UI Test Automation still exists and is enabled. |
| Build/log/artifact/test read | Request the missing Azure DevOps project/build/test permission; do not weaken evidence requirements. |
PipelinePreview |
Confirm the branch exists, the probe module is valid, and the identity can use/queue pipeline UI Test Automation. No run was created. |
No az devops configure defaults, PAT, AZURE_DEVOPS_EXT_PAT, service connection secret, or local
credential file is required. Every helper call supplies organization/project explicitly and obtains
the Azure DevOps token from the existing Azure CLI cache. Rerun preflight after switching accounts,
after token-cache changes, or immediately after any 401/403 response.