When running a build with HCP Packer enabled, Packer attempts to push
the build status to HCP.
If the build fails, we update the status to BUILD_FAILED, and that's the
end of it.
If however the build succeeds, Packer attempts to get the HCP artifact
from the builder, which will only succeed if the builder supports it.
Otherwise, we'll get either nil, or an artifact type that is not
compatible with what is expected for HCP support.
When either of those happens, we warn that the builder may not support
HCP Packer at all, so users are aware of the problem.
However, when the error was introduced, it only looked at the fact that
an error was produced, independently of the type of error. This caused
legitimate errors while building to be reported as potential
incompatibility between the builder and HCP, which was confusing to
users.
This commit changes this by introducing a new error type, only produced
when the artifact either is nil, or failed to be deserialised into a HCP
artifact, which lets us produce the incompatibility warning with more
accuracy.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
When setting a project id via the HCP_PROJECT_ID env the client will try to validate the project
by checking that it has an associated registry. If the project is invalid or not a valid UUID an error
will be displayed to the user
* Add comment to clarify usage of SDK loaded env. variables
HCP supports two types of service principals: Organization-level and project-level.
When a user tries to publish to an active HCP Packer registry using a plsp the client
fails when configuring the client due to a API permission error; namely plsp do not have
the permissions to query an org for a list of projects. Setting the HCP_PROJECT_ID does
not resolve the issue because the call to ListProjects is still executed.
This changes updates the client configuration params to obtain both the HCP Organization and
Project IDs that will be used for connecting to the HCP Packer registry. With this change
if a user provides a project Id via the HCP_PROJECT_ID environment variable no call to ListProjects will
be made. Instead the value will be take as is and used to create the connection. A user connecting with
a project level service principals must provide a valid HCP_PROJECT_ID in order to connect.
With HCP supporting multi-projects now, Packer needs to take it into
account when picking a project from an organisation.
This commit adds two cases:
1. multiple projects are defined, none is supplied through
HCP_PROJECT_ID: in this case we will default to the oldest project
defined for the organisation.
2. we supply HCP_PROJECT_ID: in this case, we pick the project with the
corresponding ID, and use it for publishing metadata.
In packer v1.8.5, the bucket's description was not properly set in the
bucket object we use for HCP, therefore all the buckets created by
Packer did not have their description updated.
Before the change
```
--- FAIL: TestReadFromHCLBuildBlock (0.00s)
--- FAIL: TestReadFromHCLBuildBlock/configure_bucket_using_only_hcp_packer_registry_block (0.00s)
types.bucket_test.go:380: expected the build to to have contents of hcp_packer_registry block but it does not: ®istry.Bucket{
Slug: "hcp_packer_registry-block-test",
- Description: "",
+ Description: "description from hcp_packer_registry block",
Destination: "",
BucketLabels: {"org": "test"},
... // 5 identical fields
}
FAIL
FAIL github.com/hashicorp/packer/internal/hcp/registry 1.072s
FAIL
```
After Change
```
~> go test ./...
? github.com/hashicorp/packer/internal/hcp/api [no test files]
ok github.com/hashicorp/packer/internal/hcp/env (cached)
ok github.com/hashicorp/packer/internal/hcp/registry 1.130s
```
Fingerprints are how we link a packer build to an iteration on HCP.
These are computed automatically from the Git SHA in the current state,
and are unique to the bucket/iteration.
The main problem with this approach is that while sound in theory, it
quickly falls apart when users want to run the same build configuration
twice, but expect a new image to be created.
With the current model, this fails, as the iteration with the current
SHA already exists.
While this is solvable through environment variables, or by committing a
change to the repository, we think this is not clear enough, and causes
an extra step to what should otherwise be a simple process.
Therefore, to lower the barrier of entry into HCP, we change this
behaviour with this commit.
Now, fingerprints are randomly generated ULIDs instead of a git SHA, and
a new one is always generated, unless one is already specified in the
environment.
This makes continuation of an existing iteration a conscious choice
rather than something automatic, and virtually eliminates conflicts such
as the ones described above.
The iteration structure that we use for linking a packer build to an
iteration on HCP defines a `Labels' attribute, which is never set nor
read from at any point.
Since it is unused, we remove it in this commit.
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
Related to the previous behaviour problem, this changes how the status
of a build is set on HCP Packer, as a cancelled build is different from
a failed one, we can set it to the appropriate state on cancellation.
As a follow-up to tracking the template type for an iteration, we
actively check that it remains consistent between two invocations of
packer, and reject the update if it is not the case.
In order to ensure users keep the same templates between two packer
invocations for a single iteration, we start tracking the type of
template used when building images.
In JSON templates, we register the builds to process on HCP at the
initialisation of the Registry.
However, during the HCP breakout, this code was duplicated with a slight
difference for the JSON templates, and the build names were registered
once during initialisation, with the expected full name, and once at the
start of PopulateIteration, with the truncated name.
This caused a bug when processing a template that contains a builder
with both a name and a type, as both would be registered, but only one
would complete, leading to the iteration being unable to complete.
The ancestry inferral code relied on HCP datasource outputs for deciding
what would be the ancestry for an image built with HCP.
This brings a dependency into the hcp package, which is not really
necessary as we only need a few information from those entities, hence
this commit removes the full dependency on the structures, in favour of
more focused data structures where we only cherry pick what is necessary
for this code.
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
In order to determine whether or not a build is done or needs
processing, we check 3 things:
1. that its status is not done
2. that its ID is not empty
3. that it has no images linked to it
The was the image check was phrased indicated that we checked for the
reverse, so we reverse the naming to make it less confusing.
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
Previously, we'd get the fingerprint for an iteration when creating the
bucket.
Since we moved the remainder of the logic for getting the iteration to a
separate Initialize method, we also move the logic that reads the
fingerprint from the environment to this function.
Fingerprint initialize was previously occurring during the creation of a
bucket. We want to be able to initialize a bucket and defer the setting
of a fingerprint to a later point.
* Update test to reflect new function signatures for Bucket and Iteration
Loadgin the slug from the environment within the Bucket implied that the
Bucket was successfully created, which may fail outside of a Git
directory when the iteration fingerprint is not set through environment
variables.
To make it possible to print as much information regarding the
environment to the users at once rather than stopping immediately when
this step fails, we move the logic to read the default value from the
environment to the HCP setup code.
The HCP_PACKER_REGISTRY environment variable had its behaviour changed
recently, as prior versions of Packer expected the attribute to be
forcefully set to a value that is neither "off" nor "0" in order to
get HCP integration to work, or that a "hcp_packer_registry" block was
defined in an HCL template. Now, this environment variable defaults to
not explicitely enable, but instead to explicitely disable HCP
integration, and the feature switch fall upon HCP_PACKER_BUCKET_NAME.
As an extra feature, we keep the prior behaviour alive when it is
explicitely defined as a value to enable it. That way we can report
errors if the rest is not defined, rather than silently ignore it.
This function we add to env is the first stone to enable this behaviour.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The current behaviour for HCP integration is based on the presence of
the HCP_PACKER_REGISTRY environment variable, where if it is either not
"0" or "off", the HCP integration is enabled.
This commit changes the behaviour to only use this variable to
explicitely disable HCP, and the HCP_PACKER_BUCKET_NAME variable will
condition if the HCP integration is enabled for a build or not in JSON
templates.
For HCL templates, the integration can also be disabled if the
HCP_PACKER_REGISTRY variable is set to "0" or "off", and will be
considered enabled if the HCP_PACKER_BUCKET_NAME is set, or there is a
"hcp_packer_registry" block in a build.
* registry: add heartbeat call for running builds
When a build is running, we send periodic heartbeats to HCP Packer's
API.
This will be used on the service side to detect if a build is stalled on
the core side because of a crash or any other malfunction that caused
Packer not to send an update on the status of a build.
* registry: only update status on status update
Prior to this commit, we'd send updates to both the labels and the
cloud-provider whenever an update to the status of a build would be
sent.
This would cause a bug in which once a build reached the post-processing
state, and its cloud-provider was set, the status could not be updated
anymore, as the cloud-provider would be set and further updates are
rejected by the platform.
To avoid this problem, we only transmit the status when doing a status
update, and no other fields along with it.
* internal: publicise CompleteBuild function
The markBuildComplete private function used to be called from a final
status update to DONE, through the UpdateBuildStatus function.
The problem being that the (now) CompleteBuild function not only updates
the status of the build, but also all its metadata.
For consistency, we remove the indirection, and explicitely call
CompleteBuild when we want to finish a build.
* internal: block status updates on DONE builds
* Add support for data source lookup
* Update failing test
* Fix typo for HCP_PACKER_BUILD_FINGERPRINT env in test
* Add test Fingerprint where needed; failing tests were not showing
because they are properly set for acctest
* Update to use BuildContext
* Update HCP Packer build labels argument
Previously when running a partial build on multi-cloud build template it
was found that build labels were only being applied at the creation for
the partially executed build. Leaving all other completed builds with
no HCP Packer build labels. This updates how incomplete builds are
loaded from the registry and ensure that any defined
hcp_packer_registry.build_labels are assigned to the build before
starting an actual Packer build.
Related to: #11573
* Add test case for overwriting build labels
* Update tests to call CreateInitialBuild for non-existing builds
* Rename test case to TestBucket_PopulateIteration
* Fix data race in PopulateIteration against mock service
Before Change
```
WARNING: DATA RACE
Write at 0x00c0005421b0 by goroutine 47:
github.com/hashicorp/packer/internal/registry.(*MockPackerClientService).PackerServiceCreateBuild()
/Users/scrubbed/Development/packer/internal/registry/mock_service.go:173 +0x2b6
github.com/hashicorp/packer/internal/registry.(*Client).CreateBuild()
/Users/scrubbed/Development/packer/internal/registry/service.go:169 +0x592
github.com/hashicorp/packer/internal/registry.(*Bucket).CreateInitialBuildForIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:107 +0x204
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration.func1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:368 +0x14e
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration·dwrap·1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:376 +0x58
Previous write at 0x00c0005421b0 by goroutine 46:
github.com/hashicorp/packer/internal/registry.(*MockPackerClientService).PackerServiceCreateBuild()
/Users/scrubbed/Development/packer/internal/registry/mock_service.go:173 +0x2b6
github.com/hashicorp/packer/internal/registry.(*Client).CreateBuild()
/Users/scrubbed/Development/packer/internal/registry/service.go:169 +0x592
github.com/hashicorp/packer/internal/registry.(*Bucket).CreateInitialBuildForIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:107 +0x204
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration.func1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:368 +0x14e
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration·dwrap·1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:376 +0x58
Goroutine 47 (running) created at:
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:362 +0x5c7
github.com/hashicorp/packer/internal/registry.TestBucket_UpdateLabelsForBuild_withMultipleBuilds()
/Users/scrubbed/Development/packer/internal/registry/types.bucket_test.go:179 +0xf7
testing.tRunner()
/usr/local/go/src/testing/testing.go:1259 +0x22f
testing.(*T).Run·dwrap·21()
/usr/local/go/src/testing/testing.go:1306 +0x47
Goroutine 46 (finished) created at:
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:362 +0x5c7
github.com/hashicorp/packer/internal/registry.TestBucket_UpdateLabelsForBuild_withMultipleBuilds()
/Users/scrubbed/Development/packer/internal/registry/types.bucket_test.go:179 +0xf7
testing.tRunner()
/usr/local/go/src/testing/testing.go:1259 +0x22f
testing.(*T).Run·dwrap·21()
/usr/local/go/src/testing/testing.go:1306 +0x47
==================
```
* Add methods for managing builds on an Iteration
* Update Mock Service to be a bit more concurrent
It is a simple set of changes to make it work for the current testing
use cases. If we need to we can move the Called fields to counters or
re-architect the mock.
This change also fixes a race condition when appending to the Slice of
errs when calling PopulateIteration.
* Update test case descriptions
* Apply suggestions from code review
Only append to errs if err is not nil
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
* Fix linting issues
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
* Rename mock bucket test file
* Add failing tests for reproducing #11573
```
=== RUN TestBucket_CreateInitialBuildForIteration
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_CreateInitialBuildForIteration (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
types.bucket_test.go:87: expected the initial build to have an additional build label but thee is no diff: ""
--- FAIL: TestBucket_UpdateLabelsForBuild (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild_withMultipleBuilds
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image2
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
types.bucket_test.go:128: expected the initial build to have an additional build label but they are equal
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
types.bucket_test.go:128: expected the initial build to have an additional build label but they are equal
--- FAIL: TestBucket_UpdateLabelsForBuild_withMultipleBuilds (0.00s)
FAIL
FAIL github.com/hashicorp/packer/internal/registry 0.646s
```
* Update CreateIntialBuildForIteration to initialize each build with a new map
Previously upon creating the initial build the same map, which was
initialized for the build_labels argument was being shared across all
build images. This was causing an issue with labels being backed by the
same map for all builds. This change ensures that all builds get their
own map with any global build labels copied over during the initial creation.
Closes#11573
Passing tests with changes on branch
```
RUN TestBucket_CreateInitialBuildForIteration
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_CreateInitialBuildForIteration (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_UpdateLabelsForBuild (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild_withMultipleBuilds
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image2
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine version:1.7.0]
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:the-original-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine version:1.7.0]
--- PASS: TestBucket_UpdateLabelsForBuild_withMultipleBuilds (0.00s)
```
* Handle errors from bucket methods
* Update test cases
Initialize maps for bucket when calling NewBucketWithIteration
On a local developer machine there is a good chance the HCP_ORG_ID and
HCP_PROJECT_ID are not set since the HCP Packer client can derive this
information using the token/secret. This change initializes the location
structure with the information obtained from the client and not the
envs.
Results of tests before change with no HCP_ORG_ID or HCP_PROJECT_ID set
```
par_test.go:185: (*packer_service.PackerServiceGetIterationOK)(nil)
par_test.go:185: &{0 [] } (*models.GrpcGatewayRuntimeError) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface
--- FAIL: TestAcc_PAR_pkr_build (3.94s)
FAIL
FAIL github.com/hashicorp/packer/internal/registry/acctest 6.446s
testing: warning: no tests to run
PASS
ok github.com/hashicorp/packer/internal/registry/env 0.318s [no tests to run]
FAIL
```
Results of tests after change with no HCP_ORG_ID or HCP_PROJECT_ID set
```
2022/01/18 12:23:48 ui: --> null.example: Published metadata to HCP Packer registry packer/pkr-acctest-temp-2/iterations/01FSQ3EPPY69PMXJSTTFYDPQ5R
--- PASS: TestAcc_PAR_pkr_build (3.20s)
PASS
ok github.com/hashicorp/packer/internal/registry/acctest 5.121s
testing: warning: no tests to run
PASS
ok github.com/hashicorp/packer/internal/registry/env 0.155s [no tests to run]
```
