We previously used a loader -> BuildConfig flow to load configuration.
This commit changes most (but not all yet) flows to use the new
graph-based approach. Instead of simply recursively loading the modules,
we now need to take a stepped approach:
1. Load the root module
2. Collect the variables and their values
3. Build the configuration with the graph-based approach
Because this approach relies on different parts from different packages,
it can't easliy be done within the `configload` package. So, now we do
most of in the backend or command.
This prevents a cyclic dependency and also makes sense semantically.
The arguments package will collect the unparsed variable values and
the backendrun helpers will work to collect the values and transform
them into terraform.InputValue.
* Update remote package's Client interface to use diagnostics instead of errors
* Update all implementations of Client interface to match changes, update calling code (incl. tests) to use diags.
* Fix S3 backend test affected by making the Workspaces method return errors via diagnostics
* Address diagnostics comparison issues in test by ensuring expected diagnostics are defined in the context of the config they're triggered by
* Fix failing test case `TestBackendConfig_EC2MetadataEndpoint/envvar_invalid_mode` by making `diagnosticBase` struct comparable
* Add compile-time checks that diagnostic types fulfil interfaces
* Stop diagnosticBase implementing ComparableDiagnostic, re-add S3-specific comparer code to s3 package
* Update tests to use the S3-specific comparer again
* Fix test case missed in refactoring
* Update the backend.Backend interface to use diagnostics as return value from StateMgr method
* Fix calls to `Fatalf`
* Update Backend interface to use diagnostics for `DeleteWorkspace` and `Workspaces` methods
* Update testing helpers in backend package
* Update pluggable package to match interface changes
* Update builtin terraform provider package to match interface changes
* Update code in Meta-related files in command package; update error check logic
* Update workspace commands; update error check logic
* Update Local backend package to match interface changes
* Update Cloud backend to match interface changes
* Update Remote backend to match interface changes
* Update cos remote-state backend to match interface changes
* Update inmem remote-state backend to match interface changes
* Update pg remote-state backend to match interface changes
* Update http remote-state backend to match interface changes
* Update consul remote-state backend to match interface changes
* Update oss remote-state backend to match interface changes
* Update kubernetes remote-state backend to match interface changes
* Update gcs remote-state backend to match interface changes
* Update s3 remote-state backend to match interface changes
* Update oci remote-state backend to match interface changes
* Update azure remote-state backend to match interface changes
* Allow warnings from Workspaces and DeleteWorkspace methods to be logged or returned to calling code
* Fix defect in inmem backend test
* Change how warnings from workspace-related methods are output to view/ui
* Fix copy-pasta
* Update some comments clarifying backend-related interfaces and "enhanced" versus "operations"
* Fix more comments that refer to types and interfaces that have moved into the backendrun package
* Update comments to reflect that the backend.Enhanced interface no longer exists
* Add compile-time check that Local backend implements backendrun.OperationsBackend
Some updates to staticcheck were catching more errors. Most of these
were format functions without a format string, but rather than adding
more exceptions I just fixed them all. This did also catch some
incorrectly wrapped errors.
We previously had all of the types and helpers for all kinds of backends
together in package backend. That kept things relatively simple, but it
also meant that the majority of backends that only deal with remote state
storage ended up still indirectly depending on the entire Terraform modules
runtime, configuration loader, etc, etc, which brings into scope a bunch
of external dependencies that the remote state backends don't really need.
Since backends that support operations are a rare exception, we'll move the
types and helpers for those into a separate package "backendrun", and
then the main package backend can have a much more modest set of types and,
more importantly, a modest set of dependencies on other packages in this
codebase.
This is part of an ongoing effort to reduce the exposure of Terraform Core
and CLI code to the remote backends and vice-versa, so that in the long
run we can more often treat them as separate for dependency maintenance
purposes.
For a very long time we've had an annoying discrepancy between the
in-memory state model and our state snapshot format where the in-memory
format stores output values for all modules whereas the snapshot format
only tracks the root module output values because those are all we
actually need to preserve between runs.
That design wart was a result of us using the state both as an internal
and an external artifact, due to having nowhere else to store the
transient values of non-root module output values while Terraform Core
does its work.
We now have namedvals.State to internally track all of the throwaway
results from named values that don't need to persist between runs, so now
we'll use that for our internal work instead and reserve the states.State
model only for the data that we will preserve between runs in state
snapshots.
The namedvals internal model isn't really designed to support enumerating
all of the output values for a particular module call, but our expression
evaluator currently depends on being able to do that and so we have a
temporary inefficient implementation of that which just scans the entire
table of values as a stopgap just to avoid this commit growing even larger
than it already is. In a future commit we'll rework the evaluator to
support the PartialEval mode and at the same time move the responsiblity
for enumerating all of the output values into the evaluator itself, since
it should be able to determine what it's expecting by analyzing the
configuration rather than just by trusting that earlier evaluation has
completed correctly.
Because our legacy state string serialization previously included output
values for all modules, some of our context tests were accidentally
depending on the implementation detail of how those got stored internally.
Those tests are updated here to test only the data that is a real part
of Terraform Core's result, by ensuring that the relevant data appears
somewhere either in a root output value or in a resource attribute.
It displays a run header with link to web UI, like starting a new plan does, then confirms the run
and streams the apply logs. If you can't apply the run (it's from a different workspace, is in an
unconfirmable state, etc. etc.), it displays an error instead.
Notable points along the way:
* Implement `WrappedPlanFile` sum type, and update planfile consumers to use it instead of a plain `planfile.Reader`.
* Enable applying a saved cloud plan
* Update TFC mocks — add org name to workspace, and minimal support for includes on MockRuns.ReadWithOptions.
Previously, remote and cloud backends would automatically alias localterraform.com as the configured hostname during configuration. This turned out to be an issue with how backends could potentially be used within the builtin terraform_remote_state data source. Those data sources each configure the same service discovery with different targets for localterraform.com, and do so simultaneously, creating an occasional concurrent map read & write panic when multiple data sources are defined.
localterraform.com is obviously not useful for every backend configuration. Therefore, I relocated the alias configuration to the callers, so they may specify when to use it. The modified design adds a new method to backend.Enhanced to allow configurators to ask which aliases should be defined.
We've seen some concern about the additional storage usage implied by
creating intermediate state snapshots for particularly long apply phases
that can arise when managing a large number of resource instances together
in a single workspace.
This is an initial coarse approach to solving that concern, just restoring
the original behavior when running inside Terraform Cloud or Enterprise
for now and not creating snapshots at all.
This is here as a solution of last resort in case we cannot find a better
compromise before the v1.5.0 final release. Hopefully a future commit
will implement a more subtle take on this which still gets some of the
benefits when running in a Terraform Enterprise environment but in a way
that will hopefully be less concerning for Terraform Enterprise
administrators.
This does not affect any other state storage implementation except the
Terraform Cloud integration and the "remote" backend's state storage when
running inside a TFC/TFE-driven remote execution environment.
The cloud backend, which communicates with TFC like APIs, can create
runs which may have one more configuration parameters altered. These
alterations are emitted as run-events on the run so that API clients
can consume and display them to users. This commit adds a step in
plan operation to query the run-events once a run is created and then
emit specific run-event descriptions to the console as warnings for
the user.
* fix: local variables should not be overridden by remote variables during `terraform import`
* chore: applied the same fix in the 'internal/cloud' package
* backport changes from cloud package to remote package
Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Co-authored-by: uturunku1 <luces.huayhuaca@gmail.com>
For Terraform Cloud users using the 'remote' backend, the existing
'pattern' prompt should work just fine - but because their workspaces
are already present in TFC, the 'migration' here is really just
realigning their local workspaces with Terraform Cloud. Instead of
forcing users to do the mental gymnastics of what it means to migrate
from 'prefix' - and because their remote workspaces probably already exist and
already conform to Terraform Cloud's naming concerns - streamline the
process for them and calculate the necessary pattern to migrate as-is,
without any user intervention necessary.
Implementing this test was quite a rabbithole, as in order to satisfy
backendTestBackendStates() the workspaces returned from
backend.Workspaces() must match exactly, and the shortcut taken to test
pagination in 3cc58813f0 created an
impossible circumstance that got plastered over with the fact that
prefix filtering is done clientside, not by the API as it should be.
Tagging does not rely on clientside filtering, and expects that the
request made to the TFC API returns exactly those workspaces with the
given tags.
These changes include a better way to test pagination, wherein we
actually create over a page worth of valid workspaces in the mock client
and implement a simplified pagination behavior to match how the TFC API
actually works.
These changes include additions to fulfill the interface for the client
mock, plus moving all that logic (which needn't be duplicated across
both the remote and cloud packages) over to the cloud package under a
dedicated mock client file.
Previously we would reject attempts to delete a workspace if its state
contained any resources at all, even if none of the resources had any
resource instance objects associated with it.
Nowadays there isn't any situation where the normal Terraform workflow
will leave behind resource husks, and so this isn't as problematic as it
might've been in the v0.12 era, but nonetheless what we actually care
about for this check is whether there might be any remote objects that
this state is tracking, and for that it's more precise to look for
non-nil resource instance objects, rather than whole resources.
This also includes some adjustments to our error messaging to give more
information about the problem and to use terminology more consistent with
how we currently talk about this situation in our documentation and
elsewhere in the UI.
We were also using the old State.HasResources method as part of some of
our tests. I considered preserving it to avoid changing the behavior of
those tests, but the new check seemed close enough to the intent of those
tests that it wasn't worth maintaining this method that wouldn't be used
in any main code anymore. I've therefore updated those tests to use
the new HasResourceInstanceObjects method instead.
