The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
Previously, we'd get the fingerprint for an iteration when creating the
bucket.
Since we moved the remainder of the logic for getting the iteration to a
separate Initialize method, we also move the logic that reads the
fingerprint from the environment to this function.
Fingerprint initialize was previously occurring during the creation of a
bucket. We want to be able to initialize a bucket and defer the setting
of a fingerprint to a later point.
* Update test to reflect new function signatures for Bucket and Iteration
Loadgin the slug from the environment within the Bucket implied that the
Bucket was successfully created, which may fail outside of a Git
directory when the iteration fingerprint is not set through environment
variables.
To make it possible to print as much information regarding the
environment to the users at once rather than stopping immediately when
this step fails, we move the logic to read the default value from the
environment to the HCP setup code.
The HCP_PACKER_REGISTRY environment variable had its behaviour changed
recently, as prior versions of Packer expected the attribute to be
forcefully set to a value that is neither "off" nor "0" in order to
get HCP integration to work, or that a "hcp_packer_registry" block was
defined in an HCL template. Now, this environment variable defaults to
not explicitely enable, but instead to explicitely disable HCP
integration, and the feature switch fall upon HCP_PACKER_BUCKET_NAME.
As an extra feature, we keep the prior behaviour alive when it is
explicitely defined as a value to enable it. That way we can report
errors if the rest is not defined, rather than silently ignore it.
This function we add to env is the first stone to enable this behaviour.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The current behaviour for HCP integration is based on the presence of
the HCP_PACKER_REGISTRY environment variable, where if it is either not
"0" or "off", the HCP integration is enabled.
This commit changes the behaviour to only use this variable to
explicitely disable HCP, and the HCP_PACKER_BUCKET_NAME variable will
condition if the HCP integration is enabled for a build or not in JSON
templates.
For HCL templates, the integration can also be disabled if the
HCP_PACKER_REGISTRY variable is set to "0" or "off", and will be
considered enabled if the HCP_PACKER_BUCKET_NAME is set, or there is a
"hcp_packer_registry" block in a build.
* registry: add heartbeat call for running builds
When a build is running, we send periodic heartbeats to HCP Packer's
API.
This will be used on the service side to detect if a build is stalled on
the core side because of a crash or any other malfunction that caused
Packer not to send an update on the status of a build.
* registry: only update status on status update
Prior to this commit, we'd send updates to both the labels and the
cloud-provider whenever an update to the status of a build would be
sent.
This would cause a bug in which once a build reached the post-processing
state, and its cloud-provider was set, the status could not be updated
anymore, as the cloud-provider would be set and further updates are
rejected by the platform.
To avoid this problem, we only transmit the status when doing a status
update, and no other fields along with it.
* internal: publicise CompleteBuild function
The markBuildComplete private function used to be called from a final
status update to DONE, through the UpdateBuildStatus function.
The problem being that the (now) CompleteBuild function not only updates
the status of the build, but also all its metadata.
For consistency, we remove the indirection, and explicitely call
CompleteBuild when we want to finish a build.
* internal: block status updates on DONE builds
* Add support for data source lookup
* Update failing test
* Fix typo for HCP_PACKER_BUILD_FINGERPRINT env in test
* Add test Fingerprint where needed; failing tests were not showing
because they are properly set for acctest
* Update to use BuildContext
* Update HCP Packer build labels argument
Previously when running a partial build on multi-cloud build template it
was found that build labels were only being applied at the creation for
the partially executed build. Leaving all other completed builds with
no HCP Packer build labels. This updates how incomplete builds are
loaded from the registry and ensure that any defined
hcp_packer_registry.build_labels are assigned to the build before
starting an actual Packer build.
Related to: #11573
* Add test case for overwriting build labels
* Update tests to call CreateInitialBuild for non-existing builds
* Rename test case to TestBucket_PopulateIteration
* Fix data race in PopulateIteration against mock service
Before Change
```
WARNING: DATA RACE
Write at 0x00c0005421b0 by goroutine 47:
github.com/hashicorp/packer/internal/registry.(*MockPackerClientService).PackerServiceCreateBuild()
/Users/scrubbed/Development/packer/internal/registry/mock_service.go:173 +0x2b6
github.com/hashicorp/packer/internal/registry.(*Client).CreateBuild()
/Users/scrubbed/Development/packer/internal/registry/service.go:169 +0x592
github.com/hashicorp/packer/internal/registry.(*Bucket).CreateInitialBuildForIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:107 +0x204
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration.func1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:368 +0x14e
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration·dwrap·1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:376 +0x58
Previous write at 0x00c0005421b0 by goroutine 46:
github.com/hashicorp/packer/internal/registry.(*MockPackerClientService).PackerServiceCreateBuild()
/Users/scrubbed/Development/packer/internal/registry/mock_service.go:173 +0x2b6
github.com/hashicorp/packer/internal/registry.(*Client).CreateBuild()
/Users/scrubbed/Development/packer/internal/registry/service.go:169 +0x592
github.com/hashicorp/packer/internal/registry.(*Bucket).CreateInitialBuildForIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:107 +0x204
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration.func1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:368 +0x14e
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration·dwrap·1()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:376 +0x58
Goroutine 47 (running) created at:
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:362 +0x5c7
github.com/hashicorp/packer/internal/registry.TestBucket_UpdateLabelsForBuild_withMultipleBuilds()
/Users/scrubbed/Development/packer/internal/registry/types.bucket_test.go:179 +0xf7
testing.tRunner()
/usr/local/go/src/testing/testing.go:1259 +0x22f
testing.(*T).Run·dwrap·21()
/usr/local/go/src/testing/testing.go:1306 +0x47
Goroutine 46 (finished) created at:
github.com/hashicorp/packer/internal/registry.(*Bucket).PopulateIteration()
/Users/scrubbed/Development/packer/internal/registry/types.bucket.go:362 +0x5c7
github.com/hashicorp/packer/internal/registry.TestBucket_UpdateLabelsForBuild_withMultipleBuilds()
/Users/scrubbed/Development/packer/internal/registry/types.bucket_test.go:179 +0xf7
testing.tRunner()
/usr/local/go/src/testing/testing.go:1259 +0x22f
testing.(*T).Run·dwrap·21()
/usr/local/go/src/testing/testing.go:1306 +0x47
==================
```
* Add methods for managing builds on an Iteration
* Update Mock Service to be a bit more concurrent
It is a simple set of changes to make it work for the current testing
use cases. If we need to we can move the Called fields to counters or
re-architect the mock.
This change also fixes a race condition when appending to the Slice of
errs when calling PopulateIteration.
* Update test case descriptions
* Apply suggestions from code review
Only append to errs if err is not nil
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
* Fix linting issues
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
* Rename mock bucket test file
* Add failing tests for reproducing #11573
```
=== RUN TestBucket_CreateInitialBuildForIteration
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_CreateInitialBuildForIteration (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
types.bucket_test.go:87: expected the initial build to have an additional build label but thee is no diff: ""
--- FAIL: TestBucket_UpdateLabelsForBuild (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild_withMultipleBuilds
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image
2022/02/16 16:22:21 [TRACE] creating initial build for component happycloud.image2
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
types.bucket_test.go:128: expected the initial build to have an additional build label but they are equal
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
types.bucket_test.go:128: expected the initial build to have an additional build label but they are equal
--- FAIL: TestBucket_UpdateLabelsForBuild_withMultipleBuilds (0.00s)
FAIL
FAIL github.com/hashicorp/packer/internal/registry 0.646s
```
* Update CreateIntialBuildForIteration to initialize each build with a new map
Previously upon creating the initial build the same map, which was
initialized for the build_labels argument was being shared across all
build images. This was causing an issue with labels being backed by the
same map for all builds. This change ensures that all builds get their
own map with any global build labels copied over during the initial creation.
Closes#11573
Passing tests with changes on branch
```
RUN TestBucket_CreateInitialBuildForIteration
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_CreateInitialBuildForIteration (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
--- PASS: TestBucket_UpdateLabelsForBuild (0.00s)
=== RUN TestBucket_UpdateLabelsForBuild_withMultipleBuilds
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image
2022/02/16 16:37:40 [TRACE] creating initial build for component happycloud.image2
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:another-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine version:1.7.0]
types.bucket_test.go:125: Comparing component build labels: map[based_off:alpine source_image:the-original-happycloud-image version:1.7.0]
against global build labels: map[based_off:alpine version:1.7.0]
--- PASS: TestBucket_UpdateLabelsForBuild_withMultipleBuilds (0.00s)
```
* Handle errors from bucket methods
* Update test cases
Initialize maps for bucket when calling NewBucketWithIteration
On a local developer machine there is a good chance the HCP_ORG_ID and
HCP_PROJECT_ID are not set since the HCP Packer client can derive this
information using the token/secret. This change initializes the location
structure with the information obtained from the client and not the
envs.
Results of tests before change with no HCP_ORG_ID or HCP_PROJECT_ID set
```
par_test.go:185: (*packer_service.PackerServiceGetIterationOK)(nil)
par_test.go:185: &{0 [] } (*models.GrpcGatewayRuntimeError) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface
--- FAIL: TestAcc_PAR_pkr_build (3.94s)
FAIL
FAIL github.com/hashicorp/packer/internal/registry/acctest 6.446s
testing: warning: no tests to run
PASS
ok github.com/hashicorp/packer/internal/registry/env 0.318s [no tests to run]
FAIL
```
Results of tests after change with no HCP_ORG_ID or HCP_PROJECT_ID set
```
2022/01/18 12:23:48 ui: --> null.example: Published metadata to HCP Packer registry packer/pkr-acctest-temp-2/iterations/01FSQ3EPPY69PMXJSTTFYDPQ5R
--- PASS: TestAcc_PAR_pkr_build (3.20s)
PASS
ok github.com/hashicorp/packer/internal/registry/acctest 5.121s
testing: warning: no tests to run
PASS
ok github.com/hashicorp/packer/internal/registry/env 0.155s [no tests to run]
```
* Add basic support for build_labels argument
* Update support for build_labels configuration argument
* Update complete test-fixture with a build_labels configuration
* Add test for deprecated labels argument
* Add deprecation for hcp_packer_registry.labels
When using the now deprecated labels argument of the new bucket_labels a
Warning will be presented to the user.
```
~> HCP_PACKER_BUILD_FINGERPRINT=356786543567865456789656789 packer
build source.pkr.hcl
Warning: the argument hcp_packer_registry.labels has been deprecated and will be removed in a future release; please use hcp_packer_registry.bucket_labels
```
When trying to use both bucket_labels and labels together an error is
presented to the user.
```
~> HCP_PACKER_BUILD_FINGERPRINT=ss6786543567865456789656789 packer
build source.pkr.hcl
Error: hcp_packer_registry.labels and hcp_packer_registry.bucket_labels are mutely exclusive; please use the recommended argument hcp_packer_registry.bucket_labels
on source.pkr.hcl line 17:
(source code not available)
```
* Update documentation for build_labels
* Apply suggestions from code review
Co-authored-by: Adrien Delorme <azr@users.noreply.github.com>
* Update hcl2template/types.build.hcp_packer_registry.go
Co-authored-by: Adrien Delorme <azr@users.noreply.github.com>
* Update handling of registry artifacts
This change uses the github.com/hashicorp/packer-plgin-sdk/packer/registryimage for querying
Artifact State for HCP Registry Image metadata. To handle the conversion
of the RPC response, mapstructure was introduced to conversion state
data into an registryimage.Image before publishing to a image bucket.
* Update to use registry image from packersdk
* Rename internal registry service pkg
* Update vendored plugins to latest version
* The latest release of Amazon, GoogleCompute, and Azure have support
for publishing images to the HCP Packer registry.
* Add working registry pkg
* Add custom error for handling the loading of PAR environment variables
* Working Publish to Build, with proper error handling for bucket names
* Update hcp-sdk-go to use branch instead of mod replace directive
* Update Packer build status configuration
* Add support for HCP_PACKER_BUILD_FINGERPRINT env
* Add support for publishing one or more PARtifacts from a single build
* add git shas to this branch
* Add ability to set provider name if available
* Add working RegistryBuilder type
* Add RegistryPostProcessor as wrapper post-processor
* When in PAR mode a empty RegistryPostProcessor is added to the end of
the post-processor list to publish all final image data.
* Add support for updating a build from PAR that is not in a DONE state
* Fix a small issue with creation the initial builds for an empty
iteration.
* Add PAR URL to post-processor display
* Implement hcp_packer_registry block (#11168)
* Update vendored Amazon plugin to v1.0.1-dev
* Fix panic when running a Packer registry build in a clean directory
* Remove the publishing of post-processor metadata from the registry
post-processor.
* Remove metadata add from registry_builder
* Update registry builder to skip a build that was found to be DONE
Co-authored-by: Megan Marsh <megan@hashicorp.com>
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
