Files
PowerToys/.pipelines/v2/templates/steps-build-installer.yml

209 lines
9.3 KiB
YAML
Raw Normal View History

Rewrite the entire Azure DevOps build system (#34984) This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in PowerToys being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. It's the same story as Terminal (https://github.com/microsoft/terminal/pull/15808). The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Builds all of the installers. - Optionally signs the output (all of the output). - Admittedly has a lot going on. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-publish-symbols-using-symbolrequestprod-api` - Consumes `**/*.pdb` from all prior build phases. - Uploads all PDBs in one artifact to Azure DevOps - Uses Microsoft's internal symbol publication REST API to submit stripped symbols to MSDL for public consumption. Finally, this pull request has some additional benefits: - Symbols are published to the private and public feeds at the same time, in the same step. They should be available in the public symbol server for public folks to debug against! - We have all the underpinnings necessary to run tests on ARM64 build agents. - Right now, `ScreenResolutionUtility` is broken - I had to introduce a custom version of `UseDotNet` which would install the right architecture (🤦); see https://github.com/microsoft/azure-pipelines-tasks/issues/20300. - All dotnet and nuget versioning is consolidated into a small set of step templates. - This will provide a great place for us to handle versioning changes later, since all versioning happens in one place.
2024-09-25 11:23:58 -05:00
parameters:
- name: versionNumber
type: string
default: "0.0.1"
- name: buildUserInstaller
type: boolean
default: false
- name: codeSign
type: boolean
default: false
- name: signingIdentity
type: object
default: {}
- name: additionalBuildOptions
type: string
default: ''
steps:
- pwsh: |-
& git clean -xfd -e *exe -- .\installer\
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Clean installer to reduce cross-contamination
- pwsh: |-
$IsPerUser = $${{ parameters.buildUserInstaller }}
$InstallerBuildSlug = "MachineSetup"
$InstallerBasename = "PowerToysSetup"
If($IsPerUser) {
$InstallerBuildSlug = "UserSetup"
$InstallerBasename = "PowerToysUserSetup"
}
$InstallerBasename += "-${{ parameters.versionNumber }}-$(BuildPlatform)"
Write-Host "##vso[task.setvariable variable=InstallerBuildSlug]$InstallerBuildSlug"
Write-Host "##vso[task.setvariable variable=InstallerRelativePath]$(BuildPlatform)\$(BuildConfiguration)\$InstallerBuildSlug"
Write-Host "##vso[task.setvariable variable=InstallerBasename]$InstallerBasename"
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Prepare Installer variables
# This dll needs to be built and signed before building the MSI.
- task: VSBuild@1
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Build PowerToysSetupCustomActions
inputs:
solution: "**/installer/PowerToysSetup.sln"
vsVersion: 17.0
msbuildArgs: >-
/t:PowerToysSetupCustomActions
/p:RunBuildEvents=true;PerUser=${{parameters.buildUserInstaller}};RestorePackagesConfig=true;CIBuild=true
-restore -graph
/bl:$(LogOutputDirectory)\installer-$(InstallerBuildSlug)-actions.binlog
${{ parameters.additionalBuildOptions }}
platform: $(BuildPlatform)
configuration: $(BuildConfiguration)
clean: true
msbuildArchitecture: x64
maximumCpuCount: true
- ${{ if eq(parameters.codeSign, true) }}:
- template: steps-esrp-signing.yml
parameters:
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Sign PowerToysSetupCustomActions
signingIdentity: ${{ parameters.signingIdentity }}
inputs:
FolderPath: 'installer/PowerToysSetupCustomActions/$(InstallerRelativePath)'
signType: batchSigning
batchSignPolicyFile: '$(build.sourcesdirectory)\.pipelines\ESRPSigning_installer.json'
ciPolicyFile: '$(build.sourcesdirectory)\.pipelines\CIPolicy.xml'
## INSTALLER START
#### MSI BUILDING AND SIGNING
- task: VSBuild@1
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Build MSI
inputs:
solution: "**/installer/PowerToysSetup.sln"
vsVersion: 17.0
msbuildArgs: >-
-restore
/t:PowerToysInstaller
/p:RunBuildEvents=false;PerUser=${{parameters.buildUserInstaller}};BuildProjectReferences=false;CIBuild=true
/bl:$(LogOutputDirectory)\installer-$(InstallerBuildSlug)-msi.binlog
${{ parameters.additionalBuildOptions }}
platform: $(BuildPlatform)
configuration: $(BuildConfiguration)
clean: false # don't undo our hard work above by deleting the CustomActions dll
msbuildArchitecture: x64
maximumCpuCount: true
- script: |-
"C:\Program Files (x86)\WiX Toolset v3.14\bin\dark.exe" -x $(build.sourcesdirectory)\extractedMsi installer\PowerToysSetup\$(InstallerRelativePath)\$(InstallerBasename).msi
dir $(build.sourcesdirectory)\extractedMsi
displayName: "${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Extract and verify MSI"
Add the Command Palette module (#37908) Windows Command Palette ("CmdPal") is the next iteration of PowerToys Run. With extensibility at its core, the Command Palette is your one-stop launcher to start _anything_. By default, CmdPal is bound to <kbd>Win+Alt+Space</kbd>. ![cmdpal-pr-002](https://github.com/user-attachments/assets/5077ec04-1009-478a-92d6-0a30989d44ac) ![cmdpal-pr-003](https://github.com/user-attachments/assets/63b4762a-9c19-48eb-9242-18ea48240ba0) ---- This brings the current preview version of CmdPal into the upstream PowerToys repo. There are still lots of bugs to work out, but it's reached the state we're ready to start sharing it with the world. From here, we can further collaborate with the community on the features that are important, and ensuring that we've got a most robust API to enable developers to build whatever extensions they want. Most of the built-in PT Run modules have already been ported to CmdPal's extension API. Those include: * Installed apps * Shell commands * File search (powered by the indexer) * Windows Registry search * Web search * Windows Terminal Profiles * Windows Services * Windows settings There are a couple new extensions built-in * You can now search for packages on `winget` and install them right from the palette. This also powers searching for extensions for the palette * The calculator has an entirely new implementation. This is currently less feature complete than the original PT Run one - we're looking forward to updating it to be more complete for future ingestion in Windows * "Bookmarks" allow you to save shortcuts to files, folders, and webpages as top-level commands in the palette. We've got a bunch of other samples too, in this repo and elsewhere ### PowerToys specific notes CmdPal will eventually graduate out of PowerToys to live as its own application, which is why it's implemented just a little differently than most other modules. Enabling CmdPal will install its `msix` package. The CI was minorly changed to support CmdPal version numbers independent of PowerToys itself. It doesn't make sense for us to start CmdPal at v0.90, and in the future, we want to be able to rev CmdPal independently of PT itself. Closes #3200, closes #3600, closes #7770, closes #34273, closes #36471, closes #20976, closes #14495 ----- TODOs et al **Blocking:** - [ ] Images and descriptions in Settings and OOBE need to be properly defined, as mentioned before - [ ] Niels is on it - [x] Doesn't start properly from PowerToys unless the fix PR is merged. - https://github.com/zadjii-msft/PowerToys/pull/556 merged - [x] I seem to lose focus a lot when I press on some limits, like between the search bar and the results. - This is https://github.com/zadjii-msft/PowerToys/issues/427 - [x] Turned off an extension like Calculator and it was still working. - Need to get rid of that toggle, it doesn't do anything currently - [x] `ListViewModel.<FetchItems>` crash - Pretty confident that was fixed in https://github.com/zadjii-msft/PowerToys/pull/553 **Not blocking / improvements:** - Show the shortcut through settings, as mentioned before, or create a button that would open CmdPalette settings. - When PowerToys starts, CmdPalette is always shown if enabled. That's weird when just starting PowerToys/ logging in to the computer with PowerToys auto-start activated. I think this should at least be a setting. - Needing to double press a result for it to do the default action seems quirky. If one is already selected, I think just pressing should be enough for it to do the action. - This is currently a setting, though we're thinking of changing the setting even more: https://github.com/zadjii-msft/PowerToys/issues/392 - There's no URI extension. Was surprised when typing a URL that it only proposed a web search. - [x] There's no System commands extension. Was expecting to be able to quickly restart the computer by typing restart but it wasn't there. - This is in PR https://github.com/zadjii-msft/PowerToys/pull/452 --------- Co-authored-by: joadoumie <98557455+joadoumie@users.noreply.github.com> Co-authored-by: Jordi Adoumie <jordiadoumie@microsoft.com> Co-authored-by: Mike Griese <zadjii@gmail.com> Co-authored-by: Niels Laute <niels.laute@live.nl> Co-authored-by: Michael Hawker <24302614+michael-hawker@users.noreply.github.com> Co-authored-by: Stefan Markovic <57057282+stefansjfw@users.noreply.github.com> Co-authored-by: Seraphima <zykovas91@gmail.com> Co-authored-by: Jaime Bernardo <jaime@janeasystems.com> Co-authored-by: Kristen Schau <47155823+krschau@users.noreply.github.com> Co-authored-by: Eric Johnson <ericjohnson327@gmail.com> Co-authored-by: Ethan Fang <ethanfang@microsoft.com> Co-authored-by: Yu Leng (from Dev Box) <yuleng@microsoft.com> Co-authored-by: Clint Rutkas <clint@rutkas.com>
2025-03-19 03:39:57 -05:00
# Extract CmdPal msix package to check if its content is signed
- pwsh: |-
Write-Host "Extracting CmdPal MSIX package"
# Define the directory to search
$searchDir = "extractedMsi\File"
# Define the regex pattern for MSIX files
$pattern = '^Microsoft.CmdPal.UI.*\.msix$'
# Get all files in the directory and subdirectories
$msixFile = Get-ChildItem -Path $searchDir -Recurse -File | Where-Object {
$_.Name -match $pattern
}
Write-Host "MSIX file found: " $msixFile
$destinationDir = "$(build.sourcesdirectory)\extractedMsi\File\extractedCmdPalMsix"
Expand-Archive -Path $msixFile -DestinationPath $destinationDir
Get-ChildItem -Path $destinationDir -Recurse -File
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Extract CmdPal MSIX package
Rewrite the entire Azure DevOps build system (#34984) This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in PowerToys being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. It's the same story as Terminal (https://github.com/microsoft/terminal/pull/15808). The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Builds all of the installers. - Optionally signs the output (all of the output). - Admittedly has a lot going on. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-publish-symbols-using-symbolrequestprod-api` - Consumes `**/*.pdb` from all prior build phases. - Uploads all PDBs in one artifact to Azure DevOps - Uses Microsoft's internal symbol publication REST API to submit stripped symbols to MSDL for public consumption. Finally, this pull request has some additional benefits: - Symbols are published to the private and public feeds at the same time, in the same step. They should be available in the public symbol server for public folks to debug against! - We have all the underpinnings necessary to run tests on ARM64 build agents. - Right now, `ScreenResolutionUtility` is broken - I had to introduce a custom version of `UseDotNet` which would install the right architecture (🤦); see https://github.com/microsoft/azure-pipelines-tasks/issues/20300. - All dotnet and nuget versioning is consolidated into a small set of step templates. - This will provide a great place for us to handle versioning changes later, since all versioning happens in one place.
2024-09-25 11:23:58 -05:00
# Check if deps.json files don't reference different dll versions.
- pwsh: |-
& '.pipelines/verifyDepsJsonLibraryVersions.ps1' -targetDir '$(build.sourcesdirectory)\extractedMsi\File'
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Audit deps.json in MSI extracted files
- ${{ if eq(parameters.codeSign, true) }}:
- pwsh: |-
& .pipelines/versionAndSignCheck.ps1 -targetDir '$(build.sourcesdirectory)\extractedMsi\File'
& .pipelines/versionAndSignCheck.ps1 -targetDir '$(build.sourcesdirectory)\extractedMsi\Binary'
git clean -xfd ./extractedMsi
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Verify all binaries are signed and versioned
- template: steps-esrp-signing.yml
parameters:
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Sign MSI
signingIdentity: ${{ parameters.signingIdentity }}
inputs:
FolderPath: 'installer/PowerToysSetup/$(InstallerRelativePath)'
signType: batchSigning
batchSignPolicyFile: '$(build.sourcesdirectory)\.pipelines\ESRPSigning_installer.json'
ciPolicyFile: '$(build.sourcesdirectory)\.pipelines\CIPolicy.xml'
#### END MSI
#### BOOTSTRAP BUILDING AND SIGNING
- task: VSBuild@1
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Build Bootstrapper
inputs:
solution: "**/installer/PowerToysSetup.sln"
vsVersion: 17.0
msbuildArgs: >-
/t:PowerToysBootstrapper
/p:PerUser=${{parameters.buildUserInstaller}};CIBuild=true
/bl:$(LogOutputDirectory)\installer-$(InstallerBuildSlug)-bootstrapper.binlog
-restore -graph
${{ parameters.additionalBuildOptions }}
platform: $(BuildPlatform)
configuration: $(BuildConfiguration)
clean: false # don't undo our hard work above by deleting the MSI
msbuildArchitecture: x64
maximumCpuCount: true
# The entirety of bundle unpacking/re-packing is unnecessary if we are not code signing it.
- ${{ if eq(parameters.codeSign, true) }}:
- script: |-
"C:\Program Files (x86)\WiX Toolset v3.14\bin\insignia.exe" -ib installer\PowerToysSetup\$(InstallerRelativePath)\$(InstallerBasename).exe -o installer\engine.exe
displayName: "${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Insignia: Extract Engine from Bundle"
- template: steps-esrp-signing.yml
parameters:
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Sign WiX Engine
signingIdentity: ${{ parameters.signingIdentity }}
inputs:
FolderPath: "installer"
Pattern: engine.exe
signConfigType: inlineSignParams
inlineOperation: |
[
{
"KeyCode": "CP-230012",
"OperationCode": "SigntoolSign",
"Parameters": {
"OpusName": "Microsoft",
"OpusInfo": "http://www.microsoft.com",
"FileDigest": "/fd \"SHA256\"",
"PageHash": "/NPH",
"TimeStamp": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256"
},
"ToolName": "sign",
"ToolVersion": "1.0"
},
{
"KeyCode": "CP-230012",
"OperationCode": "SigntoolVerify",
"Parameters": {},
"ToolName": "sign",
"ToolVersion": "1.0"
}
]
- script: |-
"C:\Program Files (x86)\WiX Toolset v3.14\bin\insignia.exe" -ab installer\engine.exe installer\PowerToysSetup\$(InstallerRelativePath)\$(InstallerBasename).exe -o installer\PowerToysSetup\$(InstallerRelativePath)\$(InstallerBasename).exe
displayName: "${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Insignia: Merge Engine into Bundle"
- template: steps-esrp-signing.yml
parameters:
displayName: ${{replace(replace(parameters.buildUserInstaller,'True','👤'),'False','💻')}} Sign Final Bootstrapper
signingIdentity: ${{ parameters.signingIdentity }}
inputs:
FolderPath: 'installer/PowerToysSetup/$(InstallerRelativePath)'
signType: batchSigning
batchSignPolicyFile: '$(build.sourcesdirectory)\.pipelines\ESRPSigning_installer.json'
ciPolicyFile: '$(build.sourcesdirectory)\.pipelines\CIPolicy.xml'
#### END BOOTSTRAP
## END INSTALLER