upstream/vault
1
0
Fork 0
mirror of https://github.com/hashicorp/vault.git synced 2026-02-23 01:50:18 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
vault/scripts/ci-helper.sh

199 lines
4.5 KiB
Bash
Raw Normal View History

[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
#!/usr/bin/env bash
adding copyright header (#19555) * adding copyright header * fix fmt and a test
2023-03-15 12:00:52 -04:00
# Copyright (c) HashiCorp, Inc.
[COMPLIANCE] License changes (#22290) * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License. Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * add missing license headers * Update copyright file headers to BUS-1.1 * Fix test that expected exact offset on hcl file --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com> Co-authored-by: Sarah Thompson <sthompson@hashicorp.com> Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2023-08-10 21:14:03 -04:00
# SPDX-License-Identifier: BUSL-1.1
adding copyright header (#19555) * adding copyright header * fix fmt and a test
2023-03-15 12:00:52 -04:00
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 15:46:04 -05:00
# The ci-helper is used to determine build metadata, build Vault binaries,
# package those binaries into artifacts, and execute tests with those artifacts.
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
set -euo pipefail
# We don't want to get stuck in some kind of interactive pager
export GIT_PAGER=cat
# Get the build date from the latest commit since it can be used across all
# builds
function build_date() {
# It's tricky to do an RFC3339 format in a cross platform way, so we hardcode UTC
: "${DATE_FORMAT:="%Y-%m-%dT%H:%M:%SZ"}"
git show --no-show-signature -s --format=%cd --date=format:"$DATE_FORMAT" HEAD
}
# Get the revision, which is the latest commit SHA
function build_revision() {
git rev-parse HEAD
}
# Determine our repository by looking at our origin URL
function repo() {
basename -s .git "$(git config --get remote.origin.url)"
}
# Determine the artifact basename based on metadata
function artifact_basename() {
: "${PKG_NAME:="vault"}"
: "${GOOS:=$(go env GOOS)}"
: "${GOARCH:=$(go env GOARCH)}"
[QT-637] Streamline our build pipeline (#24892) Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-02-06 16:11:33 -05:00
: "${VERSION_METADATA:="ce"}"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
: "${VERSION:=""}"
if [ -z "$VERSION" ]; then
echo "You must specify the VERSION variable for this command" >&2
exit 1
fi
[QT-637] Streamline our build pipeline (#24892) Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-02-06 16:11:33 -05:00
local version
version="$VERSION"
if [ "$VERSION_METADATA" != "ce" ]; then
version="${VERSION}+${VERSION_METADATA}"
fi
echo "${PKG_NAME}_${version}_${GOOS}_${GOARCH}"
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
}
# Bundle the dist directory into a zip
function bundle() {
: "${BUNDLE_PATH:=$(repo_root)/vault.zip}"
[QT-637] Streamline our build pipeline (#24892) Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-02-06 16:11:33 -05:00
echo "--> Bundling dist/* to $BUNDLE_PATH..."
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
zip -r -j "$BUNDLE_PATH" dist/
}
# Determine the root directory of the repository
function repo_root() {
git rev-parse --show-toplevel
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
}
# Build the UI
function build_ui() {
local repo_root
repo_root=$(repo_root)
pushd "$repo_root"
mkdir -p http/web_ui
popd
pushd "$repo_root/ui"
Sidebar Navigation (#19296) * Add Helios Design System Components (#19278) * adds hds dependency * updates reset import path * sets minifyCSS advanced option to false * Remove node-sass (#19376) * removes node-sass and fixes sass compilation * fixes active tab li class * Sidebar Navigation Components (#19446) * links ember-shared-components addon and imports styles * adds sidebar frame and nav components * updates HcNav component name to HcAppFrame and adds sidebar UserMenu component * adds tests for sidebar components * fixes tests * updates user menu styling * fixes typos in nav cluster component * changes padding value in sidebar stylesheet to use variable * Replace and remove old nav components with new ones (#19447) * links ember-shared-components addon and imports styles * adds sidebar frame and nav components * updates activeCluster on auth service and adds activeSession prop for sidebar visibility * replaces old nav components with new ones in templates * fixes sidebar visibility issue and updates user menu label class * removes NavHeader usage * adds clients index route to redirect to dashboard * removes unused HcAppFrame footer block and reduces page header top margin * Nav component cleanup (#19681) * removes nav-header components * removes navbar styling * removes status-menu component and styles * removes cluster and auth info components * removes menu-sidebar component and styling * fixes tests * Console Panel Updates (#19741) * updates console panel styling * adds test for opening and closing the console panel * updates console panel background color to use hds token * adds right margin to console panel input * updates link-status banner styling * updates hc nav components to new API * Namespace Picker Updates (#19753) * updates namespace-picker * updates namespace picker menu styling * adds bottom margin to env banner * updates class order on namespace picker link * restores manage namespaces refresh icon * removes manage namespaces nav icon * removes home link component (#20027) * Auth and Error View Updates (#19749) * adds vault logo to auth page * updates top level error template * updates loading substate handling and moves policies link from access to cluster nav (#20033) * moves console panel to bottom of viewport (#20183) * HDS Sidebar Nav Components (#20197) * updates nav components to hds * upgrades project yarn version to 3.5 * fixes issues in app frame component * updates sidenav actions to use icon button component * Sidebar navigation acceptance tests (#20270) * adds sidebar navigation acceptance tests and fixes other test failures * console panel styling tweaks * bumps addon version * remove and ignore yarn install-state file * fixes auth service and console tests * moves classes from deleted files after bulma merge * fixes sass syntax errors blocking build * cleans up dart sass deprecation warnings * adds changelog entry * hides namespace picker when sidebar nav panel is minimized * style tweaks * fixes sidebar nav tests * bumps hds addon to latest version and removes style override * updates modify-passthrough-response helper * updates sidebar nav tests * mfa-setup test fix attempt * fixes cluster mfa setup test * remove deprecated yarn ignore-optional flag from makefile * removes another instance of yarn ignore-optional and updates ui readme * removes unsupported yarn verbose flag from ci-helper * hides nav headings when user does not have access to any sub links * removes unused optional deps and moves lint-staged to dev deps * updates has-permission helper and permissions service tests * fixes issue with console panel not filling container width
2023-05-02 21:36:15 -04:00
yarn install
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
npm rebuild node-sass
Sidebar Navigation (#19296) * Add Helios Design System Components (#19278) * adds hds dependency * updates reset import path * sets minifyCSS advanced option to false * Remove node-sass (#19376) * removes node-sass and fixes sass compilation * fixes active tab li class * Sidebar Navigation Components (#19446) * links ember-shared-components addon and imports styles * adds sidebar frame and nav components * updates HcNav component name to HcAppFrame and adds sidebar UserMenu component * adds tests for sidebar components * fixes tests * updates user menu styling * fixes typos in nav cluster component * changes padding value in sidebar stylesheet to use variable * Replace and remove old nav components with new ones (#19447) * links ember-shared-components addon and imports styles * adds sidebar frame and nav components * updates activeCluster on auth service and adds activeSession prop for sidebar visibility * replaces old nav components with new ones in templates * fixes sidebar visibility issue and updates user menu label class * removes NavHeader usage * adds clients index route to redirect to dashboard * removes unused HcAppFrame footer block and reduces page header top margin * Nav component cleanup (#19681) * removes nav-header components * removes navbar styling * removes status-menu component and styles * removes cluster and auth info components * removes menu-sidebar component and styling * fixes tests * Console Panel Updates (#19741) * updates console panel styling * adds test for opening and closing the console panel * updates console panel background color to use hds token * adds right margin to console panel input * updates link-status banner styling * updates hc nav components to new API * Namespace Picker Updates (#19753) * updates namespace-picker * updates namespace picker menu styling * adds bottom margin to env banner * updates class order on namespace picker link * restores manage namespaces refresh icon * removes manage namespaces nav icon * removes home link component (#20027) * Auth and Error View Updates (#19749) * adds vault logo to auth page * updates top level error template * updates loading substate handling and moves policies link from access to cluster nav (#20033) * moves console panel to bottom of viewport (#20183) * HDS Sidebar Nav Components (#20197) * updates nav components to hds * upgrades project yarn version to 3.5 * fixes issues in app frame component * updates sidenav actions to use icon button component * Sidebar navigation acceptance tests (#20270) * adds sidebar navigation acceptance tests and fixes other test failures * console panel styling tweaks * bumps addon version * remove and ignore yarn install-state file * fixes auth service and console tests * moves classes from deleted files after bulma merge * fixes sass syntax errors blocking build * cleans up dart sass deprecation warnings * adds changelog entry * hides namespace picker when sidebar nav panel is minimized * style tweaks * fixes sidebar nav tests * bumps hds addon to latest version and removes style override * updates modify-passthrough-response helper * updates sidebar nav tests * mfa-setup test fix attempt * fixes cluster mfa setup test * remove deprecated yarn ignore-optional flag from makefile * removes another instance of yarn ignore-optional and updates ui readme * removes unsupported yarn verbose flag from ci-helper * hides nav headings when user does not have access to any sub links * removes unused optional deps and moves lint-staged to dev deps * updates has-permission helper and permissions service tests * fixes issue with console panel not filling container width
2023-05-02 21:36:15 -04:00
yarn run build
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
popd
}
# Build Vault
function build() {
local revision
local build_date
local ldflags
local msg
# Get or set our basic build metadata
revision=$(build_revision)
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
build_date=$(build_date) #
: "${BIN_PATH:="dist/"}" #if not run by actions-go-build (enos local) then set this explicitly
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
: "${GO_TAGS:=""}"
Keep symbols by default (#20519) By reversing the logic and adding a `REMOVE_SYMBOLS` environment variable that, when set, will remove symbols. This has been requested to re-enable Dynatrace support, which requires symbols are intact. Sadly this increases the size (on my mac) from 192,609,682 bytes to 236,696,722 bytes (+23% increase). I confirmed that this adds symbols back, and that `dlv` will load the Vault binary.
2023-05-04 16:23:06 -04:00
: "${REMOVE_SYMBOLS:=""}"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
[QT-506] Use enos scenario samples for testing (#22641) Replace our prior implementation of Enos test groups with the new Enos sampling feature. With this feature we're able to describe which scenarios and variant combinations are valid for a given artifact and allow enos to create a valid sample field (a matrix of all compatible scenarios) and take an observation (select some to run) for us. This ensures that every valid scenario and variant combination will now be a candidate for testing in the pipeline. See QT-504[0] for further details on the Enos sampling capabilities. Our prior implementation only tested the amd64 and arm64 zip artifacts, as well as the Docker container. We now include the following new artifacts in the test matrix: * CE Amd64 Debian package * CE Amd64 RPM package * CE Arm64 Debian package * CE Arm64 RPM package Each artifact includes a sample definition for both pre-merge/post-merge (build) and release testing. Changes: * Remove the hand crafted `enos-run-matrices` ci matrix targets and replace them with per-artifact samples. * Use enos sampling to generate different sample groups on all pull requests. * Update the enos scenario matrices to handle HSM and FIPS packages. * Simplify enos scenarios by using shared globals instead of cargo-culted locals. Note: This will require coordination with vault-enterprise to ensure a smooth migration to the new system. Integrating new scenarios or modifying existing scenarios/variants should be much smoother after this initial migration. [0] https://github.com/hashicorp/enos/pull/102 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-09-08 14:46:32 -04:00
(unset GOOS; unset GOARCH; go generate ./...)
Prevent breakage when merging OSS changes to ENT due to missing funcs (#21564) Add a go:generate helper called stubmaker, which generates appropriate stubs on ent based on oss stubs, but only when needed (i.e. real ent funcs haven't been added yet.)
2023-07-06 12:35:06 -04:00
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
# Build our ldflags
[QT-637] Streamline our build pipeline (#24892) Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-02-06 16:11:33 -05:00
msg="--> Building Vault revision $revision, built $build_date..."
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
Add symbols back to the build to fix Dynatrace support (#20294)
2023-04-21 13:15:56 -04:00
# Keep the symbol and dwarf information by default
Keep symbols by default (#20519) By reversing the logic and adding a `REMOVE_SYMBOLS` environment variable that, when set, will remove symbols. This has been requested to re-enable Dynatrace support, which requires symbols are intact. Sadly this increases the size (on my mac) from 192,609,682 bytes to 236,696,722 bytes (+23% increase). I confirmed that this adds symbols back, and that `dlv` will load the Vault binary.
2023-05-04 16:23:06 -04:00
if [ -n "$REMOVE_SYMBOLS" ]; then
Add symbols back to the build to fix Dynatrace support (#20294)
2023-04-21 13:15:56 -04:00
ldflags="-s -w "
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
else
Keep symbols by default (#20519) By reversing the logic and adding a `REMOVE_SYMBOLS` environment variable that, when set, will remove symbols. This has been requested to re-enable Dynatrace support, which requires symbols are intact. Sadly this increases the size (on my mac) from 192,609,682 bytes to 236,696,722 bytes (+23% increase). I confirmed that this adds symbols back, and that `dlv` will load the Vault binary.
2023-05-04 16:23:06 -04:00
ldflags=""
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
fi
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
ldflags="${ldflags} -X github.com/hashicorp/vault/version.GitCommit=$revision -X github.com/hashicorp/vault/version.BuildDate=$build_date"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
Onboard Vault to CRT version bump automation (#18311) * adding new version bump refactoring * address comments * remove changes used for testing * add the version bump event! * fix local enos scenarios * remove unnecessary local get_local_metadata steps from scenarios * add version base, pre, and meta to the get_local_metadata module * use the get_local_metadata module in the local builder for version metadata * update the version verifier to always require a build date Signed-off-by: Ryan Cragun <me@ryan.ec> * Update to embed the base version from the VERSION file directly into version.go. This ensures that any go tests can use the same (valid) version as CI and so can local builds and local enos runs. We still want to be able to set a default metadata value in version_base.go as this is not something that we set in the VERSION file - we pass this in as an ldflag in CI (matters more for ENT but we want to keep these files in sync across repos). * update comment * fixing bad merge * removing actions-go-build as it won't work with the latest go caching changes * fix logic for getting version in enos-lint.yml * fix version number * removing unneeded module --------- Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Claire <claire@hashicorp.com> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-06 12:08:48 -04:00
if [[ ${VERSION_METADATA+x} ]]; then
msg="${msg}, metadata ${VERSION_METADATA}"
ldflags="${ldflags} -X github.com/hashicorp/vault/version.VersionMetadata=$VERSION_METADATA"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
fi
# Build vault
echo "$msg"
pushd "$(repo_root)"
mkdir -p dist
mkdir -p out
set -x
[QT-637] Streamline our build pipeline (#24892) Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-02-06 16:11:33 -05:00
go env
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
go build -v -tags "$GO_TAGS" -ldflags "$ldflags" -o dist/
set +x
popd
}
legal: include license in release zip and docker image (#26801) * legal: include license in release zip and docker image * Move license logic to script * Add cp license to build vault action * test * Trigger Build
2024-05-17 11:18:38 -04:00
# ENT: Prepare legal requirements for packaging
function prepare_ent_legal() {
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
: "${PKG_NAME:="vault"}"
pushd "$(repo_root)"
mkdir -p dist
curl -o dist/EULA.txt https://eula.hashicorp.com/EULA.txt
curl -o dist/TermsOfEvaluation.txt https://eula.hashicorp.com/TermsOfEvaluation.txt
mkdir -p ".release/linux/package/usr/share/doc/$PKG_NAME"
cp dist/EULA.txt ".release/linux/package/usr/share/doc/$PKG_NAME/EULA.txt"
cp dist/TermsOfEvaluation.txt ".release/linux/package/usr/share/doc/$PKG_NAME/TermsOfEvaluation.txt"
popd
}
legal: include license in release zip and docker image (#26801) * legal: include license in release zip and docker image * Move license logic to script * Add cp license to build vault action * test * Trigger Build
2024-05-17 11:18:38 -04:00
# CE: Prepare legal requirements for packaging
function prepare_ce_legal() {
: "${PKG_NAME:="vault"}"
pushd "$(repo_root)"
mkdir -p dist
cp LICENSE dist/LICENSE.txt
mkdir -p ".release/linux/package/usr/share/doc/$PKG_NAME"
cp LICENSE ".release/linux/package/usr/share/doc/$PKG_NAME/LICENSE.txt"
popd
}
[QT-506] Use enos scenario samples for testing (#22641) Replace our prior implementation of Enos test groups with the new Enos sampling feature. With this feature we're able to describe which scenarios and variant combinations are valid for a given artifact and allow enos to create a valid sample field (a matrix of all compatible scenarios) and take an observation (select some to run) for us. This ensures that every valid scenario and variant combination will now be a candidate for testing in the pipeline. See QT-504[0] for further details on the Enos sampling capabilities. Our prior implementation only tested the amd64 and arm64 zip artifacts, as well as the Docker container. We now include the following new artifacts in the test matrix: * CE Amd64 Debian package * CE Amd64 RPM package * CE Arm64 Debian package * CE Arm64 RPM package Each artifact includes a sample definition for both pre-merge/post-merge (build) and release testing. Changes: * Remove the hand crafted `enos-run-matrices` ci matrix targets and replace them with per-artifact samples. * Use enos sampling to generate different sample groups on all pull requests. * Update the enos scenario matrices to handle HSM and FIPS packages. * Simplify enos scenarios by using shared globals instead of cargo-culted locals. Note: This will require coordination with vault-enterprise to ensure a smooth migration to the new system. Integrating new scenarios or modifying existing scenarios/variants should be much smoother after this initial migration. [0] https://github.com/hashicorp/enos/pull/102 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-09-08 14:46:32 -04:00
# Package version converts a vault version string into a compatible representation for system
# packages.
function version_package() {
awk '{ gsub("-","~",$1); print $1 }' <<< "$VAULT_VERSION"
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 15:46:04 -05:00
}
# Run the CI Helper
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
function main() {
case $1 in
artifact-basename)
artifact_basename
;;
build)
build
;;
build-ui)
build_ui
;;
bundle)
bundle
;;
date)
build_date
;;
legal: include license in release zip and docker image (#26801) * legal: include license in release zip and docker image * Move license logic to script * Add cp license to build vault action * test * Trigger Build
2024-05-17 11:18:38 -04:00
prepare-ent-legal)
prepare_ent_legal
;;
prepare-ce-legal)
prepare_ce_legal
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
;;
revision)
build_revision
;;
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 15:46:04 -05:00
version-package)
version_package
;;
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 15:14:43 -05:00
*)
echo "unknown sub-command" >&2
exit 1
;;
esac
}
main "$@"
Reference in a new issue Copy permalink