mirror of
https://github.com/microsoft/PowerToys.git
synced 2026-08-29 10:09:43 +02:00
[UITests][Keyboard Manager] UI tests for new keyboard manager. (#50059)
<!-- Suggested title: [UITests][Keyboard Manager] Add and stabilize UI tests for the unified editor --> ## Summary of the Pull Request Closes: #40662 Adds a new `Microsoft.PowerToys.UITest.Next` end-to-end suite for Keyboard Manager and hardens the unified editor behavior uncovered while exercising it on Windows 10, Windows 11, x64, and ARM64. The Keyboard Manager suite executes 32 test cases covering: - Unified editor create, edit, save, enable/disable, delete, restart persistence, missing-profile recovery, validation, and special actions. - Single-key, key-to-shortcut, shortcut-to-shortcut, shortcut-to-key, disabled-key, and app-specific remapping. - Modifier ordering and release behavior, including Alt+Tab and Alt+F4 targets. - Real keyboard behavior through Windows Notepad and Calculator fixtures rather than synthetic input windows. The PR also adds 26 managed unit tests around settings normalization, profile reconciliation, metadata ownership, active-state changes, and canonical modifier ordering. ## Detailed Description of the Pull Request / Additional comments ### Keyboard Manager UI tests Adds `src/modules/keyboardmanager/Tests/KeyboardManager.UITests` as a Microsoft Testing Platform executable using `UITestAutomation.Next` and winappcli. The suite includes: - `KeyboardManager.Editor.CreateEditPersistDelete` - Creates and edits mappings. - Verifies native profile and editor metadata persistence across restarts. - Exercises active-state toggling, row deletion, and missing native-profile recovery. - `KeyboardManager.Editor.InputAndValidation` - Covers key recording, dropdown input, keyboard navigation, cancellation, and app-specific validation. - `KeyboardManager.Editor.ActionPersistence` - Covers Open URL, Open app, and Insert text actions and verifies canonical readback after restart. - Parameterized runtime tests for key/shortcut combinations, disabled targets, app-specific mappings, modifier release order, Alt+Tab, and Alt+F4. The test support code provides: - A shared cross-process fixture lock and isolated Keyboard Manager settings scope. - Real Notepad documents with exact window/document ownership and cleanup. - Win10/Win11-aware Calculator window ownership for Alt+F4 assertions. - A low-level keyboard event recorder that validates injected key-down/key-up sequences and flags. - Authoritative persisted JSON assertions instead of relying only on transient UI state. ### Keyboard Manager correctness fixes found by the suite The tests exposed product races and persistence issues that are fixed in the same PR: - Native profile JSON is written through checked same-directory atomic replacement instead of a truncation-prone direct write. - Editor metadata and native mappings are normalized and reconciled per profile. - Create, edit, delete, and active-state mutations commit metadata and native state transactionally under a cross-process lock. - Startup reconciliation preserves inactive metadata owned by other profiles and repairs legacy/profileless settings. - Native Boolean return values use one-byte marshaling to match the C++ ABI. - Modifier keys and serialized targets are canonicalized consistently. - Unit-test initialization no longer starts real settings synchronization. ### CI stability hardening The editor workflows use coordinate-free UIA invocation for command buttons and authoritative-signal retries for idempotent row interactions. Window fixtures bind exact top-level HWNDs and distinguish the Win10 Calculator `ApplicationFrameWindow` from its child content window. The child-specific `UITestAutomation.Next` updates add or improve: - `WindowShowWatcher` lifecycle handling used by the Keyboard Manager window fixtures. - Exact HWND foreground, close, and process-tree cleanup helpers required by the Win10/Win11 tests. ### Pipeline workflow Adds the internal `ui-tests-pipeline-ci` skill and Azure DevOps helper used during stabilization: - Uses an existing Azure CLI session plus Azure DevOps REST APIs without per-call authentication prompts. - Supports paged branch/build discovery, preview and queue operations, exact-SHA reconciliation, stage retry/cancel, logs, test results, artifacts, and direct result-attachment downloads. - Uses build-scoped one-shot completion monitoring and retains the three-run stabilization guardrail. ## Validation Steps Performed ### Azure DevOps UI Test Automation Final verification build: - https://microsoft.visualstudio.com/Dart/_build/results?buildId=155512681&view=results - https://microsoft.visualstudio.com/Dart/_build/results?buildId=155525972&view=results ## Reviewer guide Suggested review order: 1. `src/modules/keyboardmanager/Tests/KeyboardManager.UITests/` - intended workflows and assertions. 2. `KeyboardManagerEditorUI/Settings/SettingsManager.cs` and `Pages/MainPage.xaml.cs` - transaction and reconciliation ownership. 3. `common/MappingConfiguration.cpp` and `Interop/KeyboardManagerInterop.cs` - native persistence and ABI fixes. 4. `src/common/UITestAutomation.Next/WindowControl.cs` and `WindowShowWatcher.cs` - child-specific window lifecycle hardening. 5. `.github/skills/ui-tests-pipeline-ci/` - internal Azure CLI/REST stabilization workflow. Local evidence: <img width="1237" height="854" alt="image" src="https://github.com/user-attachments/assets/ff49652e-fbc8-4581-a48f-836dbed37b6e" />
This commit is contained in:
Gleb Khmyznikov
416
.github/skills/ui-tests-pipeline-ci/references/agentic-loop.md
vendored
Normal file
416
.github/skills/ui-tests-pipeline-ci/references/agentic-loop.md
vendored
Normal file
@@ -0,0 +1,416 @@
|
||||
# Azure DevOps UI-test CI agentic loop
|
||||
|
||||
This is an internal post-local-validation workflow. It assumes the selected UITest project already
|
||||
passed the complete local matrix required by
|
||||
[ui-tests-local-vm](../../ui-tests-local-vm/SKILL.md), including default and constrained profiles.
|
||||
|
||||
## 0. Use the existing Azure CLI session
|
||||
|
||||
All Azure DevOps operations in this workflow use PowerShell 7, the existing Azure CLI sign-in, and
|
||||
Azure DevOps REST APIs. This avoids per-call authentication prompts and works for definitions,
|
||||
builds, timelines, logs, preview/queue, stage retry/cancel, test results, artifacts, and result
|
||||
attachments.
|
||||
|
||||
Dot-source the bundled helper and validate the session once:
|
||||
|
||||
```pwsh
|
||||
. .\.github\skills\ui-tests-pipeline-ci\scripts\AzureDevOps.ps1
|
||||
Test-AzDevOpsSession
|
||||
```
|
||||
|
||||
The helper obtains a token for Azure DevOps resource
|
||||
`499b84ac-1321-427f-aa17-267ca6975798` on each request. Azure CLI serves it from its cache and
|
||||
silently refreshes it when needed. The token and authorization header stay in memory and are cleared
|
||||
after every request. Repeated reads and actual pipeline queueing were verified without prompts on
|
||||
2026-08-20.
|
||||
|
||||
Never run `az login` through an agent, request credentials, print or persist a token/header, enable
|
||||
command tracing around authentication, or commit downloaded internal evidence. If
|
||||
`Test-AzDevOpsSession` fails, ask the user to authenticate outside the agent and stop with an access
|
||||
blocker. Do not fall back to another transport.
|
||||
|
||||
`Invoke-AzDevOpsRest` accepts a project-relative REST path and returns `{ Body, Headers }`:
|
||||
|
||||
```pwsh
|
||||
$build = (Invoke-AzDevOpsRest -Uri '_apis/build/builds/123?api-version=7.1').Body
|
||||
```
|
||||
|
||||
`Get-AzDevOpsPagedValues` follows every `x-ms-continuationtoken` response header. Use it whenever an
|
||||
endpoint can paginate; never infer completeness from one page.
|
||||
|
||||
REST writes mutate Azure state. Queue, cancel, retry, or approve only when the user's CI request
|
||||
authorizes that action. The absence of a per-call authentication dialog is not authorization.
|
||||
|
||||
## 1. Prove the local gate
|
||||
|
||||
Before touching Azure DevOps, record:
|
||||
|
||||
- Exact pushed branch and commit SHA.
|
||||
- Clean x64 and ARM64 builds where applicable.
|
||||
- Complete suite on Windows 10 and Windows 11 under the default profile.
|
||||
- Complete suite under `Constrained` (1 vCPU, 4 GB).
|
||||
- Windows 11 ARM64 guest evidence on a Windows-on-ARM host when applicable.
|
||||
- Zero skipped, inconclusive, or not-executed tests and zero export errors.
|
||||
|
||||
Do not substitute a focused run for full sign-off. If a required local environment is unavailable,
|
||||
stop and ask the user before consuming CI.
|
||||
|
||||
Derive `uiTestModules` from the exact `.csproj` filename without `.csproj`, for example:
|
||||
|
||||
```text
|
||||
FancyZonesEditor.UITests.Next
|
||||
```
|
||||
|
||||
The list must be non-empty and contain only the projects currently being changed.
|
||||
|
||||
## 2. Discover the pipeline and serialize the branch
|
||||
|
||||
Discover rather than assuming IDs:
|
||||
|
||||
```pwsh
|
||||
$branch = 'refs/heads/<current-branch>'
|
||||
$definitionName = [Uri]::EscapeDataString('UI Test Automation')
|
||||
$definitions = (Invoke-AzDevOpsRest -Uri `
|
||||
"_apis/build/definitions?name=$definitionName&api-version=7.1").Body.value
|
||||
$enabled = @($definitions | Where-Object queueStatus -EQ 'enabled')
|
||||
if ($enabled.Count -ne 1) {
|
||||
throw "Expected one enabled UI Test Automation definition; found $($enabled.Count)."
|
||||
}
|
||||
$pipelineId = [int]$enabled[0].id
|
||||
|
||||
$encodedBranch = [Uri]::EscapeDataString($branch)
|
||||
$history = Get-AzDevOpsPagedValues -Uri `
|
||||
"_apis/build/builds?definitions=$pipelineId&branchName=$encodedBranch&queryOrder=queueTimeDescending&%24top=100&api-version=7.1"
|
||||
$active = @($history.Items | Where-Object status -IN @('notStarted', 'inProgress', 'postponed', 'cancelling'))
|
||||
```
|
||||
|
||||
Require every returned build's `sourceBranch` to equal the exact target ref. Normally only one run
|
||||
for that branch may be active. Other branches do not block queueing and must not be canceled.
|
||||
|
||||
Adopt an active run only when its source SHA, selected platforms, modules, `buildSource`, and reused
|
||||
build ID exactly match the checkpoint. A mismatching same-branch run is a blocker. Wait, or cancel
|
||||
only a superseded run that the user owns or explicitly asked to stop:
|
||||
|
||||
```pwsh
|
||||
Invoke-AzDevOpsRest `
|
||||
-Uri "_apis/build/builds/${buildId}?api-version=7.1" `
|
||||
-Method Patch `
|
||||
-Body @{ status = 'cancelling' }
|
||||
```
|
||||
|
||||
Confirm it becomes terminal `completed`/`canceled`; `cancelling` still occupies the branch slot.
|
||||
|
||||
A narrow exception allows parallel same-branch execution only when the user explicitly authorizes a
|
||||
supplemental run for an architecture whose product build failed while another architecture remains
|
||||
active. Use the same pushed SHA and modules, select only the failed architecture, and checkpoint both
|
||||
build IDs. The supplemental run remains part of the attempt ledger and never erases the original
|
||||
failure.
|
||||
|
||||
## 3. Choose `buildNow` or `specificBuildId`
|
||||
|
||||
| Situation | `buildSource` | `specificBuildId` |
|
||||
|---|---|---|
|
||||
| First run in the sequence | `buildNow` | `xxxx` |
|
||||
| Product/runtime/common/pipeline files changed | `buildNow` | `xxxx` |
|
||||
| Prior product build failed, was incomplete, or lacks one selected-platform artifact | `buildNow` | `xxxx` |
|
||||
| Only selected UITest project files changed and every selected product artifact succeeded | `specificBuildId` | Prior numeric build ID |
|
||||
|
||||
For reuse, prove all of the following:
|
||||
|
||||
1. Previous target-branch run is terminal.
|
||||
2. Product build stages succeeded for every selected platform.
|
||||
3. Artifact names include normal `build-x64-Release` and/or `build-arm64-Release` artifacts as
|
||||
selected. `build-<platform>-Release-failure-<attempt>` is diagnostic and never reusable.
|
||||
4. `git diff --name-only <previous-sourceVersion>..HEAD` is confined to selected UITest project
|
||||
directories. Any shared, product, pipeline, dependency, or installer change requires `buildNow`.
|
||||
5. The revision is pushed.
|
||||
|
||||
Use numeric build IDs such as `154921069`, never display numbers such as `20260814.4`.
|
||||
|
||||
## 4. Preview and queue
|
||||
|
||||
Default parameters:
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| `buildPlatforms` | `- arm64\n- x64` |
|
||||
| `enableMsBuildCaching` | `false` |
|
||||
| `useVSPreview` | `false` |
|
||||
| `useLatestWebView2` | `false` |
|
||||
| `buildSource` | Decision from section 3 |
|
||||
| `specificBuildId` | `xxxx` or prior numeric build ID as a string |
|
||||
| `uiTestModules` | Bracketed non-empty list, e.g. `[KeyboardManager.UITests]` |
|
||||
|
||||
Do not silently remove a platform. `x64` expands to Windows 10 and Windows 11 jobs; `arm64` expands
|
||||
to the ARM64 job.
|
||||
|
||||
Preview first with the exact branch and string-valued template parameters:
|
||||
|
||||
```pwsh
|
||||
$templateParameters = @{
|
||||
buildPlatforms = "- arm64`n- x64"
|
||||
enableMsBuildCaching = 'false'
|
||||
useVSPreview = 'false'
|
||||
useLatestWebView2 = 'false'
|
||||
buildSource = 'buildNow'
|
||||
specificBuildId = 'xxxx'
|
||||
uiTestModules = '[KeyboardManager.UITests]'
|
||||
}
|
||||
$request = @{
|
||||
previewRun = $true
|
||||
resources = @{ repositories = @{ self = @{ refName = $branch } } }
|
||||
templateParameters = $templateParameters
|
||||
}
|
||||
$preview = (Invoke-AzDevOpsRest `
|
||||
-Uri "_apis/pipelines/${pipelineId}/runs?api-version=7.1-preview.1" `
|
||||
-Method Post `
|
||||
-Body $request).Body
|
||||
```
|
||||
|
||||
Inspect `finalYaml`. Require only the expected `Build_<platform>` and test stages, exact module
|
||||
assignments, no unrequested platform, and no prior build ID for `buildNow`. Preview creates no real
|
||||
build and does not consume an attempt.
|
||||
|
||||
Immediately repeat the full branch preflight from section 2. If clear, queue by changing only:
|
||||
|
||||
```pwsh
|
||||
$request.previewRun = $false
|
||||
$run = (Invoke-AzDevOpsRest `
|
||||
-Uri "_apis/pipelines/${pipelineId}/runs?api-version=7.1-preview.1" `
|
||||
-Method Post `
|
||||
-Body $request).Body
|
||||
```
|
||||
|
||||
Record `id`, `name`, web link, branch, resolved repository version, and echoed parameters. Query the
|
||||
returned build ID and require `sourceVersion` to equal the pushed SHA. If it differs, cancel before
|
||||
tests and stop.
|
||||
|
||||
Queue and active-run discovery are not atomic. Immediately list every active exact-branch run again,
|
||||
following all continuation tokens. The oldest matching run owns the normal branch slot. Cancel a
|
||||
younger run created by this agent when safe; never cancel someone else's run. Every actual queue,
|
||||
including a reconciled duplicate, counts in the ledger.
|
||||
|
||||
### Persist the checkpoint
|
||||
|
||||
Store in session/task state, never the repository:
|
||||
|
||||
```text
|
||||
Pipeline: UI Test Automation / <definition ID>
|
||||
Build: <build ID> / <build number> / <web link>
|
||||
Branch: <exact refs/heads/... ref>
|
||||
Source SHA: <40-character commit>
|
||||
Attempt: <n>/3
|
||||
Build source: <buildNow|specificBuildId> [reused build ID]
|
||||
Platforms: <selected platforms>
|
||||
Modules: [<exact project stems>]
|
||||
State: <notStarted|inProgress|...>
|
||||
```
|
||||
|
||||
After any interruption, scheduled wake, or user turn, query every exact checkpointed build ID before
|
||||
other Azure work. Never rediscover by adopting the newest run.
|
||||
|
||||
### One-hour scheduled continuation
|
||||
|
||||
Do not leave a nonterminal build with a passive handoff. Full builds normally take 60-90 minutes and
|
||||
test-only validation about 40 minutes. Arm one one-shot host wake-up at a time; after each wake,
|
||||
query exact build IDs and re-arm for one hour only if work remains nonterminal.
|
||||
|
||||
Monitoring state is scoped to exact build IDs. If the user reports a tracked build's status and asks
|
||||
to stop its now-obsolete watcher, remove only that build's task/marker. Do not treat that as a
|
||||
standing opt-out: every later build queued by the agent must immediately receive a fresh one-shot
|
||||
continuation and remain monitored until terminal unless the user explicitly opts out for that new
|
||||
build.
|
||||
|
||||
The scheduled action only wakes the agent. Do not put `az`, a token, build parameters, or any Azure
|
||||
request in it. On Windows without a native agent scheduler:
|
||||
|
||||
1. Persist build IDs, unique task name, marker under `$env:TEMP`, scheduled UTC/local time, and
|
||||
watcher terminal ID in session state.
|
||||
2. Register a current-user, limited one-shot task for `(Get-Date).AddHours(1)` whose only action
|
||||
writes the marker. Use an encoded PowerShell command for deterministic quoting.
|
||||
3. Start one asynchronous `FileSystemWatcher` terminal for the marker. Do not use `Start-Sleep`, a
|
||||
timer loop, or repeated Azure requests.
|
||||
4. On notification, remove marker and task, then query exact IDs through `Invoke-AzDevOpsRest`.
|
||||
5. Re-arm only after an authenticated status query confirms a build remains nonterminal.
|
||||
|
||||
## 5. Monitor exact builds
|
||||
|
||||
Use these endpoints through `Invoke-AzDevOpsRest`:
|
||||
|
||||
| Evidence | REST path |
|
||||
|---|---|
|
||||
| Build status/source | `_apis/build/builds/<BUILD_ID>?api-version=7.1` |
|
||||
| Stage/job timeline and issues | `_apis/build/builds/<BUILD_ID>/timeline?api-version=7.1` |
|
||||
| Log metadata | `_apis/build/builds/<BUILD_ID>/logs?api-version=7.1` |
|
||||
| Narrow log content | `_apis/build/builds/<BUILD_ID>/logs/<LOG_ID>?startLine=<N>&endLine=<N>&api-version=7.1` |
|
||||
| Pipeline artifacts | `_apis/build/builds/<BUILD_ID>/artifacts?api-version=7.1` |
|
||||
| Test runs for build | `_apis/test/runs?buildUri=vstfs%3A%2F%2F%2FBuild%2FBuild%2F<BUILD_ID>&api-version=7.1` |
|
||||
| Paged test results | `_apis/test/Runs/<RUN_ID>/results?%24top=1000&%24skip=<N>&api-version=7.1` |
|
||||
|
||||
Read build status, then timeline, newest log timestamps, relevant narrow logs, every non-passing test
|
||||
result, and artifacts. Page test results by `$skip` until fewer than 1,000 remain. Classify every
|
||||
outcome other than `Passed`, including failed, aborted, error, timeout, not-executed, inconclusive,
|
||||
blocked, warning, not-applicable, and paused.
|
||||
|
||||
Do not busy-poll. Fresh timeline/log timestamps are progress.
|
||||
|
||||
| REST status/result | Action |
|
||||
|---|---|
|
||||
| `notStarted`, `inProgress`, `postponed`, `cancelling` | Continue monitoring exact ID |
|
||||
| `completed` + `succeeded` | Verify selected tests executed and every result passed |
|
||||
| `completed` + `partiallySucceeded` | Treat as failure until warnings/results are understood |
|
||||
| `completed` + `failed` | Diagnose logs, tests, screenshots, and recordings |
|
||||
| `completed` + `canceled` | Record who/why; not a product/test failure |
|
||||
| `completed` + `abandoned` | Infrastructure/administrative termination |
|
||||
|
||||
### Retry or cancel one stage
|
||||
|
||||
Use the YAML stage key such as `Build_x64`, not its display name. Retry only a terminal failed stage:
|
||||
|
||||
```pwsh
|
||||
$stageRefName = 'Build_x64'
|
||||
Invoke-AzDevOpsRest `
|
||||
-Uri "_apis/build/builds/${buildId}/stages/${stageRefName}?api-version=7.1-preview.1" `
|
||||
-Method Patch `
|
||||
-Body @{ state = 'retry'; forceRetryAllJobs = $false }
|
||||
```
|
||||
|
||||
REST success is not proof that retry started. Re-read the timeline until the stage/job attempt
|
||||
increments and becomes pending/in-progress. Retry materialization can be delayed. Do not queue a
|
||||
supplemental run until that check establishes the update was rejected or remained a no-op.
|
||||
|
||||
Use `state = 'cancel'` to cancel one known stage. Use the build PATCH from section 2 for the whole
|
||||
run. Re-read status after every mutation.
|
||||
|
||||
## 6. Diagnose failures and collect evidence
|
||||
|
||||
First determine whether selected test stages ran. An empty failed-result query can also mean a
|
||||
product build failed and dependent tests were skipped.
|
||||
|
||||
### Product build failed before tests
|
||||
|
||||
For each selected platform:
|
||||
|
||||
1. Read stage/job timeline issues as routing data.
|
||||
2. Locate `Build Release_<platform>` and read a narrow log tail around the first `##[error]`, compiler
|
||||
error, MSBuild error summary, test failure, or nonzero exit. Report the first actionable cause,
|
||||
not the later generic task-exit message.
|
||||
3. Compare artifacts:
|
||||
- `build-<platform>-Release` is the normal product artifact.
|
||||
- `build-<platform>-Release-failure-<attempt>` is diagnostics only.
|
||||
4. Confirm dependent test stages are skipped.
|
||||
5. State that no screenshot/video exists when no test ran.
|
||||
6. Never reuse a partial product build with `specificBuildId`.
|
||||
|
||||
Download a useful failure artifact to a temporary directory with the already-installed extension:
|
||||
|
||||
```pwsh
|
||||
az pipelines runs artifact download `
|
||||
--organization https://dev.azure.com/microsoft `
|
||||
--project Dart `
|
||||
--run-id <BUILD_ID> `
|
||||
--artifact-name <FAILURE_ARTIFACT_NAME> `
|
||||
--path <TEMP_DIRECTORY> `
|
||||
--only-show-errors
|
||||
```
|
||||
|
||||
Do not install tools or commit evidence. Inspect text logs first; open `.binlog` only with an already
|
||||
available Structured Log Viewer.
|
||||
|
||||
### UI test stages ran
|
||||
|
||||
1. Query all non-passing results and preserve `(runId, resultId)` because titles repeat by platform.
|
||||
2. Resolve platform from the owning job/log, never result order.
|
||||
3. Read error, stack, duration, assembly, and `Standard_Console_Output.log`.
|
||||
4. Open failure screenshot and `recording_*.mp4` before changing code.
|
||||
5. Apply the authoritative-signal analysis from the local VM loop.
|
||||
|
||||
Construct an attachments-pane link for every failed result:
|
||||
|
||||
```text
|
||||
https://microsoft.visualstudio.com/Dart/_build/results?buildId=<BUILD_ID>&view=ms.vss-test-web.build-test-results-tab&runId=<RUN_ID>&resultId=<RESULT_ID>&paneView=attachments
|
||||
```
|
||||
|
||||
### Download result attachments without browser help
|
||||
|
||||
Azure Test result attachments are separate from pipeline artifacts. List and download them directly:
|
||||
|
||||
```pwsh
|
||||
$buildId = <BUILD_ID>
|
||||
$runId = <RUN_ID>
|
||||
$resultId = <RESULT_ID>
|
||||
$destination = Join-Path $env:TEMP "PowerToys-CI-$buildId-$runId-$resultId"
|
||||
New-Item -ItemType Directory -Path $destination -Force | Out-Null
|
||||
|
||||
$resultBase = "_apis/test/Runs/${runId}/Results/${resultId}"
|
||||
$attachments = (Invoke-AzDevOpsRest `
|
||||
-Uri "$resultBase/attachments?api-version=7.1-preview.1").Body.value
|
||||
$selected = @($attachments | Where-Object {
|
||||
$_.fileName -eq 'Standard_Console_Output.log' -or
|
||||
$_.fileName -like 'failure-*.png' -or
|
||||
$_.fileName -like 'recording_*.mp4'
|
||||
})
|
||||
if (-not ($selected.fileName -contains 'Standard_Console_Output.log')) {
|
||||
throw 'Standard_Console_Output.log is missing from the failed result.'
|
||||
}
|
||||
|
||||
foreach ($attachment in $selected) {
|
||||
$target = Join-Path $destination ([IO.Path]::GetFileName([string]$attachment.fileName))
|
||||
Invoke-AzDevOpsRest `
|
||||
-Uri "$resultBase/Attachments/$($attachment.id)?api-version=7.1-preview.1" `
|
||||
-OutFile $target | Out-Null
|
||||
[pscustomobject]@{
|
||||
AttachmentId = $attachment.id
|
||||
FileName = Split-Path $target -Leaf
|
||||
Bytes = (Get-Item $target).Length
|
||||
Sha256 = (Get-FileHash $target -Algorithm SHA256).Hash
|
||||
Path = $target
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Read the console log and relevant product logs directly, inspect visual evidence, and record bytes
|
||||
plus SHA-256. Never ask the user to fetch files that this path can download.
|
||||
|
||||
Pipeline artifact metadata contains `resource.downloadUrl`; include useful authenticated links in
|
||||
reports. Do not confuse multi-gigabyte product artifacts with Azure Test recordings.
|
||||
|
||||
## 7. Iterate, maximum three runs
|
||||
|
||||
Maintain this ledger:
|
||||
|
||||
| Attempt | Build ID / number | Source SHA | Build source | Product build | Tests | Failure signature | Evidence links | Progress |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1/3 | | | `buildNow` | | | | | Baseline |
|
||||
| 2/3 | | | | | | | | |
|
||||
| 3/3 | | | | | | | | |
|
||||
|
||||
Every actual queued build counts, including infrastructure failures and canceled duplicates. Preview
|
||||
runs do not. A supplemental architecture run also counts unless the user explicitly authorizes an
|
||||
exception after seeing the ledger.
|
||||
|
||||
Before another attempt:
|
||||
|
||||
1. State one falsifiable hypothesis from logs and visual evidence.
|
||||
2. Make the smallest relevant fix without weakening assertions.
|
||||
3. Build and rerun the focused test locally.
|
||||
4. Rerun affected full default and constrained suites.
|
||||
5. Commit and push.
|
||||
6. Confirm current run terminal or explicitly handle the supplemental-architecture exception.
|
||||
7. Re-evaluate section 3.
|
||||
|
||||
Progress means fewer failures/platforms, the original test passing, a later authoritative state, or
|
||||
a broad timeout narrowed to actionable evidence. Error-text churn is not progress. After run 3 or
|
||||
three no-progress runs, stop and ask the user unless they explicitly authorize an exception.
|
||||
|
||||
## 8. Report the result
|
||||
|
||||
Include:
|
||||
|
||||
- Build link, numeric ID, display number, pipeline ID, branch, and exact SHA.
|
||||
- Attempt number, `buildSource`, reused ID, platforms, and modules.
|
||||
- Current/terminal status and per-platform counts.
|
||||
- Every non-passing test and first actionable root-cause line.
|
||||
- One attachments link per failed result, or an explicit no-test/no-video statement.
|
||||
- Useful pipeline artifact links without confusing them with recordings.
|
||||
- Whether next action is wait, local fix, retry, success, or escalation.
|
||||
Reference in New Issue
Block a user