upstream/vault
1
0
Fork 0
mirror of https://github.com/hashicorp/vault.git synced 2026-02-03 20:40:45 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 11
vault/.github/workflows/build.yml
Ryan Cragun 06ab7d2702
[VAULT-38882] actions(build): build and test on changes to gotoolchain (#31481) 
Make sure that we always build and test the application if the go
toolchain changes.

Signed-off-by: Ryan Cragun <me@ryan.ec>
2025-08-12 15:51:05 -06:00

479 lines
25 KiB
YAML
Raw Blame History

name: build
# Some words of caution before modifying this workflow:
# This file and workflow have been carefully architected to meet the following requirements:
# * Builds and tests the correct artifacts in both CE and Ent while maintaining a merge-conflict-free
# build.yml between the two repos
# * Supports multiple Github event triggers
# * Is highly optimized for cost and speed
# * Supports a variety of complex use cases
# If you wish to modify this file/workflow, please consider:
# * That the workflow must work under when triggered by pull_request, push, schedule, and
# workflow_dispatch events.
# * Merge-conflict-free compatibility between CE and Ent. Any changes that you make here must work
# in both repository contexts.
# * There are many workflow flow control modifiers. Further details below.
# * The total number of workers and the runner size. Further details below.
# Further details:
# * The workflow is used by the CRT system for building, notarizing, signing, and releasing
# artifacts. Whatever we do in this workflow must support building all artifacts and uploading
# them to Github in order to fulfill the CRT requirement, while also maintaining a smaller
# default build matrix for the pull requests.
# * CRT is designed to trigger a workflow called build in a workflow file called build.yml. This
# file must build the correct artifacts in CE and Ent, depending on the repository context.
# We've gone to great lengths to architect this file and workflow so that we can build and test
# the correct artifacts in each context while maintaining a merge-conflict-free file between CE
# and Ent. Any changes that you make here must work in both repository contexts.
# * The workflow must support multiple event triggers, all of which have varying event payloads
# which must be considered. If you make changes you must ensure that the workflow still works
# under normal pull_request, push, schedule, and workflow_dispatch trigger events.
# * The workflow has been highly optimized for cost and speed. If possible, it's better to add a
# step to an existing job than create another job. Over a long time horizon a new job is often
# much more expensive than a single step in an existing job, they also take up a limited number
# of our available runners.
# * Flow control in the workflow is complex in order to support many various use cases, including:
# * Only building on tier 1 supported "core" artifacts by default.
# * Only building the UI if the Go application or UI has been modified.
# * Skipping builds entirely if the commit or PR only modifies changelog or website documentation.
# * The ability to check out the HEAD reference instead of a Github merge branch reference.
# * The ability to control building all of our tier 2 supported "extended" artifacts via a
# build/all label, even if the event trigger is pull_request or, more importantly, a push.
# It's important to note that we must maintain support for building all artifacts on push
# via a pull request, even though push events aren't directly tied to pull requests. Our
# label metadata helpers are designed to handle this complexity.
# * The ability to build all of our artifacts on a scheduled cadence to ensure we don't
# accidentally regress.
# * All of these considerations, and many others, have led to the modular design we have here.
# * If you're doing something in more than one place, try and use small composite actions
# whenever possible.
on:
workflow_dispatch:
pull_request:
types:
- opened
- ready_for_review
- reopened
- synchronize
push:
branches:
- main
- release/**
# TODO: VAULT-34830: - ce/**
schedule:
- cron: '05 02 * * *' # * is a special character in YAML so you have to quote this string
concurrency:
group: ${{ github.head_ref || github.run_id }}-build
cancel-in-progress: true
jobs:
setup:
# Setup is our entrypoint into the entire workflow. Here we gather metadata and export useful
# outputs for further use as inputs or for flow control.
#
# Trigger the setup workflow if any of the following conditions are true:
# * The workflow was triggered by a push (merge) to the main or release branch.
# * The workflow was triggered by pull request and the pull request is not a draft.
# * The workflow was triggered by on schedule to test building all artifacts.
if: |
github.event_name == 'push' ||
github.event_name == 'schedule' ||
(github.event_name == 'pull_request' && github.event.pull_request.draft == false)
runs-on: ${{ github.repository == 'hashicorp/vault' && 'ubuntu-latest' || fromJSON('["self-hosted","ubuntu-latest-x64"]') }}
permissions: write-all # vault-auth
outputs:
build-date: ${{ steps.metadata.outputs.vault-build-date }}
changed-files: ${{ steps.changed-files.outputs.changed-files }}
checkout-ref: ${{ steps.checkout.outputs.ref }}
compute-build: ${{ steps.metadata.outputs.compute-build }}
compute-build-ui: ${{ steps.metadata.outputs.compute-build-ui }}
compute-small: ${{ steps.metadata.outputs.compute-small }}
is-draft: ${{ steps.metadata.outputs.is-draft }}
is-ent-branch: ${{ steps.metadata.outputs.is-ent-branch }}
is-ent-repo: ${{ steps.metadata.outputs.is-ent-repo }}
is-fork: ${{ steps.metadata.outputs.is-fork }}
labels: ${{ steps.metadata.outputs.labels }}
vault-binary-name: ${{ steps.metadata.outputs.vault-binary-name }}
vault-revision: ${{ steps.metadata.outputs.vault-revision }}
vault-version: ${{ steps.metadata.outputs.vault-version }}
vault-version-metadata: ${{ steps.metadata.outputs.vault-version-metadata }}
vault-version-package: ${{ steps.metadata.outputs.vault-version-package }}
workflow-trigger: ${{ steps.metadata.outputs.workflow-trigger }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
# Make sure we check out correct ref based on PR labels and such
- uses: ./.github/actions/checkout
id: checkout
# Get the vault version metadata
- uses: hashicorp/actions-set-product-version@v2
id: set-product-version
with:
checkout: false # don't override the reference we've checked out
# Gather additional metadata about our execution context
- uses: ./.github/actions/metadata
id: metadata
with:
vault-version: ${{ steps.set-product-version.outputs.product-version }}
# Get the elevated github token
- if: steps.metadata.outputs.is-ent-repo == 'true'
id: vault-auth
name: Vault Authenticate
run: vault-auth
- if: steps.metadata.outputs.is-ent-repo == 'true'
id: vault-secrets
name: Fetch Vault Secrets
uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3.0.0
with:
url: ${{ steps.vault-auth.outputs.addr }}
caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }}
token: ${{ steps.vault-auth.outputs.token }}
secrets: |
kv/data/github/${{ github.repository }}/github-token token | ELEVATED_GITHUB_TOKEN;
# Determine the changed files
- uses: ./.github/actions/changed-files
id: changed-files
with:
github-token: ${{ steps.metadata.outputs.is-ent-repo != 'true' && secrets.ELEVATED_GITHUB_TOKEN || steps.vault-secrets.outputs.ELEVATED_GITHUB_TOKEN }}
# Ensure that we have not changed any enterprise files on pull requests against ce/* branches.
# We do this here because we have the information, there's absolutely no reason to go
# further until we've resolved the issue, and we want to fail a required workflow if this
# issue is present.
- if: |
steps.metadata.outputs.is-ent-repo == 'true' &&
steps.metadata.outputs.workflow-trigger == 'pull_request' &&
startsWith(github.event.pull_request.base.ref, 'ce/') &&
contains(fromJSON(steps.changed-files.outputs.changed-files).groups, 'enterprise')
name: Ensure that we have not changed any enterprise files on pull requests against ce/* branches.
run: |
echo "The pull request has changed files that are in enterprise groups!"
echo "If you believe this to be in error you will want to update the changed files checks in tools/pipeline/internal/pkg/changed"
echo "on our enterprise branches and backport them to ce/* before continuing with this pull request."
echo "See the 'changed-files' step above for a list of changed files and their associated metadata groups."
exit 1
# Make sure all required Go modules are cached at this point. We don't want all of the Go
# tests and build jobs to download modules and race to upload them to the cache.
- uses: ./.github/actions/set-up-go
name: Ensure Go modules are cached
with:
github-token: ${{ steps.metadata.outputs.is-ent-repo != 'true' && secrets.ELEVATED_GITHUB_TOKEN || steps.vault-secrets.outputs.ELEVATED_GITHUB_TOKEN }}
# Don't download them on a cache hit during setup, just make sure they're cached before
# subsequent workflows are run.
no-restore: true
ui:
# The Web UI workflow is a prerequisite workflow for building our artifacts. If the application
# or UI change we'll trigger this workflow but only build it if we don't already have the asset
# in our Github cache.
#
# Ensure the Web UI is built if any of the following conditions are true:
# * The workflow was triggered by a push (merge) to the main or release branch.
# * The workflow was triggered by on schedule to test building all artifacts.
# * The `build/all` tag is present on either a pull request or on the pull request that created
# a merge
# * The workflow was triggered by a pull request, the pull request is not a draft, and the UI
# or app changed.
# * Our pipeline or test scenarios changed.
if: |
needs.setup.outputs.workflow-trigger == 'push' ||
needs.setup.outputs.workflow-trigger == 'schedule' ||
contains(fromJSON(needs.setup.outputs.labels), 'build/all') ||
(
needs.setup.outputs.workflow-trigger == 'pull_request' &&
needs.setup.outputs.is-draft == 'false' &&
(
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'ui') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gotoolchain') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'pipeline') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'enos') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'app')
)
)
needs: setup
runs-on: ${{ fromJSON(needs.setup.outputs.compute-build-ui) }}
outputs:
cache-key: ui-${{ steps.ui-hash.outputs.ui-hash }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: ${{ needs.setup.outputs.checkout-ref }}
- name: Get UI hash
id: ui-hash
run: echo "ui-hash=$(git ls-tree HEAD ui --object-only)" | tee -a "$GITHUB_OUTPUT"
- name: Set up UI asset cache
id: cache-ui-assets
uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
with:
enableCrossOsArchive: true
lookup-only: true
path: http/web_ui
# Only restore the UI asset cache if we haven't modified anything in the ui directory.
# Never do a partial restore of the web_ui if we don't get a cache hit.
key: ui-${{ steps.ui-hash.outputs.ui-hash }}
- if: steps.cache-ui-assets.outputs.cache-hit != 'true'
name: Set up node and yarn
uses: actions/setup-node@cdca7365b2dadb8aad0a33bc7601856ffabcc48e # v4.3.0
with:
node-version-file: ui/package.json
cache: yarn
cache-dependency-path: ui/yarn.lock
- if: steps.cache-ui-assets.outputs.cache-hit != 'true'
name: Build UI
run: make ci-build-ui
# Artifacts is where we'll build the various Vault binaries and package them into their respective
# Zip bundles, RPM and Deb packages, and container images. After we've packaged them we upload
# them to the Github Actions artifacts storage and execute our Enos test scenarios. If the
# workflow is triggered by a push to main CRT will take these artifacts from Github and perform
# all of the necessary notarizing and signing before uploading them to Artifactory.
#
# # Trigger the setup workflow if any of the following conditions are true:
#
# * The workflow was triggered by on schedule to test building all artifacts.
# * The Go app was changed.
# * Our pipeline or test scenarios changed.
# * The build/all label is present on a pull request or push.
artifacts-ce:
if: |
needs.setup.outputs.is-ent-branch != 'true' && (
needs.setup.outputs.workflow-trigger == 'schedule' ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'app') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gotoolchain') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'pipeline') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'enos') ||
contains(fromJSON(needs.setup.outputs.labels), 'build/all')
)
needs:
- setup
- ui # Don't build and test artifacts unless the UI build was triggered.
uses: ./.github/workflows/build-artifacts-ce.yml
with:
build-all: ${{contains(fromJSON(needs.setup.outputs.labels), 'build/all') || needs.setup.outputs.workflow-trigger == 'schedule' || contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gotoolchain')}}
build-date: ${{ needs.setup.outputs.build-date }}
checkout-ref: ${{ needs.setup.outputs.checkout-ref }}
compute-build: ${{ needs.setup.outputs.compute-build }}
compute-small: ${{ needs.setup.outputs.compute-small }}
vault-revision: ${{ needs.setup.outputs.vault-revision }}
vault-version: ${{ needs.setup.outputs.vault-version }}
vault-version-package: ${{ needs.setup.outputs.vault-version-package }}
web-ui-cache-key: ${{ needs.ui.outputs.cache-key }}
secrets: inherit
artifacts-ent:
if: |
needs.setup.outputs.is-ent-branch == 'true' && (
needs.setup.outputs.workflow-trigger == 'schedule' ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'app') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gotoolchain') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'pipeline') ||
contains(fromJSON(needs.setup.outputs.changed-files).groups, 'enos') ||
contains(fromJSON(needs.setup.outputs.labels), 'build/all')
)
needs:
- setup
- ui # Don't build and test artifacts unless the UI build was triggered.
uses: ./.github/workflows/build-artifacts-ent.yml
with:
build-all: ${{contains(fromJSON(needs.setup.outputs.labels), 'build/all') || needs.setup.outputs.workflow-trigger == 'schedule' || contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gotoolchain')}}
build-date: ${{ needs.setup.outputs.build-date }}
checkout-ref: ${{ needs.setup.outputs.checkout-ref }}
compute-build: ${{ needs.setup.outputs.compute-build }}
compute-small: ${{ needs.setup.outputs.compute-small }}
vault-revision: ${{ needs.setup.outputs.vault-revision }}
vault-version: ${{ needs.setup.outputs.vault-version }}
vault-version-package: ${{ needs.setup.outputs.vault-version-package }}
web-ui-cache-key: ${{ needs.ui.outputs.cache-key }}
secrets: inherit
test:
# Test all of the testable artifacts if our repo isn't a fork. We don't test when the PR is
# created from a fork because secrets are not passed in and they are required.
if: |
always() &&
needs.setup.outputs.is-fork == 'false' &&
( needs.artifacts-ce.result == 'success' || needs.artifacts-ent.result == 'success' )
name: test ${{ matrix.artifact }}
needs:
- setup
- ui
- artifacts-ce
- artifacts-ent
uses: ./.github/workflows/test-run-enos-scenario-matrix.yml
strategy:
fail-fast: false
matrix:
include: ${{ needs.setup.outputs.is-ent-branch == 'true' && fromJSON(needs.artifacts-ent.outputs.testable-packages) || fromJSON(needs.artifacts-ce.outputs.testable-packages) }}
with:
build-artifact-name: ${{ matrix.artifact }}
runs-on: ${{ github.repository == 'hashicorp/vault' && '"ubuntu-latest"' || '["self-hosted","ubuntu-latest-x64"]' }}
sample-max: 1
sample-name: ${{ matrix.sample }}
ssh-key-name: ${{ github.event.repository.name }}-ci-ssh-key
vault-edition: ${{ matrix.edition }}
vault-revision: ${{ needs.setup.outputs.vault-revision }}
vault-version: ${{ needs.setup.outputs.vault-version-metadata }}
secrets: inherit
test-containers:
# Test all of the testable containers if our repo isn't a fork. We don't test when the PR is
# created from a fork because secrets are not passed in and they are required (for now).
if: |
always() &&
needs.setup.outputs.is-fork == 'false' &&
( needs.artifacts-ce.result == 'success' || needs.artifacts-ent.result == 'success' )
name: test ${{ matrix.artifact }}
needs:
- setup
- ui
- artifacts-ce
- artifacts-ent
uses: ./.github/workflows/test-run-enos-scenario-containers.yml
strategy:
fail-fast: false
matrix:
include: ${{ needs.setup.outputs.is-ent-branch == 'true' && fromJSON(needs.artifacts-ent.outputs.testable-containers) || fromJSON(needs.artifacts-ce.outputs.testable-containers) }}
with:
build-artifact-name: ${{ matrix.artifact }}
sample-max: 1
sample-name: ${{ matrix.sample }}
vault-edition: ${{ matrix.edition }}
vault-revision: ${{ needs.setup.outputs.vault-revision }}
vault-version: ${{ needs.setup.outputs.vault-version-metadata }}
secrets: inherit
completed-successfully:
# build/completed-successfully is the only build workflow that must pass in order to merge
# a pull request. This workflow is used to determine the overall status of all the prior
# workflows and to notify various different channels of success or failure. As part of this
# workflow we create the necessary build metadata that is required for the CRT build system.
#
# Our logic here mirrors that of setup as it and this are the only two workflows that must
# be triggered together.
if: |
always() &&
(
github.event_name == 'push' ||
github.event_name == 'schedule' ||
(github.event_name == 'pull_request' && github.event.pull_request.draft == false)
)
runs-on: ${{ github.repository == 'hashicorp/vault' && 'ubuntu-latest' || fromJSON('["self-hosted","ubuntu-latest-x64"]') }}
permissions: write-all # Ensure we have id-token:write access for vault-auth.
needs:
- setup
- ui
- artifacts-ce
- artifacts-ent
- test
- test-containers
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- id: status
name: Determine status
run: |
results=$(tr -d '\n' <<< '${{ toJSON(needs.*.result) }}')
if ! grep -q -v -E '(failure|cancelled)' <<< "$results"; then
result="failed"
else
result="success"
fi
{
echo "result=${result}"
echo "results=${results}"
} | tee -a "$GITHUB_OUTPUT"
- if: needs.setup.outputs.is-ent-repo == 'true'
id: vault-auth
name: Vault Authenticate
run: vault-auth
- if: needs.setup.outputs.is-ent-repo == 'true'
id: secrets
name: Fetch Vault Secrets
uses: hashicorp/vault-action@v3
with:
url: ${{ steps.vault-auth.outputs.addr }}
caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }}
token: ${{ steps.vault-auth.outputs.token }}
secrets: |
kv/data/github/${{ github.repository }}/github_actions_notifications_bot token | SLACK_BOT_TOKEN;
- id: slackbot-token
run:
echo "slackbot-token=${{ needs.setup.outputs.is-ent-repo != 'true' && secrets.SLACK_BOT_TOKEN || steps.secrets.outputs.SLACK_BOT_TOKEN }}" >> "$GITHUB_OUTPUT"
- if: |
needs.setup.outputs.workflow-trigger == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.event.pull_request.base.repo.full_name &&
(github.repository == 'hashicorp/vault' || github.repository == 'hashicorp/vault-enterprise')
name: Create or update a build status comment on the pull request
env:
ARTIFACTS: ${{ needs.setup.outputs.is-ent-branch == 'true' && needs.artifacts-ent.result || needs.artifacts-ce.result }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO: ${{ github.event.repository.name }}
RUN_ID: ${{ github.run_id }}
TEST: ${{ needs.test.result }}
TEST_CONTAINERS: ${{ needs.test-containers.result }}
UI: ${{ needs.ui.result }}
run: ./.github/scripts/report-build-status.sh
- name: Notify build failures in Slack
if: |
always() &&
steps.status.outputs.result != 'success' &&
(github.ref_name == 'main' || startsWith(github.ref_name, 'release/'))
uses: slackapi/slack-github-action@37ebaef184d7626c5f204ab8d3baff4262dd30f0 # v1.27.0
env:
SLACK_BOT_TOKEN: ${{ steps.slackbot-token.outputs.slackbot-token }}
with:
channel-id: "C05AABYEA9Y" # Notify #feed-vault-ci-official
# channel-id: "C05Q4D5V89W" # Notify #test-vault-ci-slack-integration
payload: |
{
"text": "${{ needs.setup.outputs.is-ent-branch == 'true' && 'vault-enterprise' || 'vault' }} build failures on ${{ github.ref_name }}",
"blocks": [
{
"type": "header",
"text": {
"type": "plain_text",
"text": ":rotating_light: ${{ needs.setup.outputs.is-ent-branch == 'true' && 'vault-enterprise' || 'vault' }} build failures on ${{ github.ref_name }} :rotating_light:",
"emoji": true
}
},
{
"type": "divider"
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${{ needs.setup.result != 'failure' && ':white_check_mark:' || ':x:' }} Setup\n${{ needs.ui.result != 'failure' && ':white_check_mark:' || ':x:' }} Build UI\n${{ (needs.setup.outputs.is-ent-branch == 'true' && needs.artifacts-ent.result != 'failure' || needs.artifacts-ce.result != 'failure' ) && ':white_check_mark:' || ':x:' }} Build Vault Artifacts\n${{ needs.test.result != 'failure' && ':white_check_mark:' || ':x:' }} Enos package test scenarios\n${{ needs.test-containers.result != 'failure' && ':white_check_mark:' || ':x:' }} Enos container test scenarios"
},
"accessory": {
"type": "button",
"text": {
"type": "plain_text",
"text": "View Failing Workflow",
"emoji": true
},
"url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
}
}
]
}
- uses: hashicorp/actions-generate-metadata@v1
# create build metadata if we successfully created artifacts
if: needs.setup.outputs.is-ent-branch == 'true' && needs.artifacts-ent.result == 'success' || needs.artifacts-ce.result == 'success'
id: generate-metadata-file
with:
version: ${{ needs.setup.outputs.vault-version-metadata }}
product: ${{ needs.setup.outputs.vault-binary-name }}
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: steps.generate-metadata-file.outcome == 'success' # upload our metadata if we created it
with:
name: metadata.json
path: ${{ steps.generate-metadata-file.outputs.filepath }}
if-no-files-found: error
- if: always() && steps.status.outputs.result != 'success'
name: Check for failed status
run: |
echo "One or more required build workflows failed: ${{ steps.status.outputs.results }}"
exit 1
Reference in a new issue View git blame Copy permalink