We previously used a loader -> BuildConfig flow to load configuration.
This commit changes most (but not all yet) flows to use the new
graph-based approach. Instead of simply recursively loading the modules,
we now need to take a stepped approach:
1. Load the root module
2. Collect the variables and their values
3. Build the configuration with the graph-based approach
Because this approach relies on different parts from different packages,
it can't easliy be done within the `configload` package. So, now we do
most of in the backend or command.
* feat: Allow reading state store configuration from a planfile and using it to prepare a Local backend that uses the state store
* test: Assert that we can get and use state store configuration from a plan file
* test: Add integration test showing that an apply command can use a plan file to configure and use a state store
* test: Add E2E test showing pluggable state storage being used with the full init-plan-apply workflow
* feat: A plan file will report the state storage provider among its required providers, if PSS is in use.
See the code comment added in this commit. This addition does not impact an apply command as the missing provider will be detected before this code is executed. However I'm making this change so that the method is still accurate is being able to return a complete list of providers needed by the plan.
* fix: Include error messages when there is a problem parsing provider or state store config when getting a backend from a planfile
* feat: Add trace logs to BackendForLocalPlan indicating when the provider is launched and the state store is configured
* chore: Small grammar change in error diagnostic
* refactor: Remove suggestions when the plan's state store doesn't match the implementations in the provider
* test: Add test coverage of what happens when the contents of a plan file using PSS doesn't match the resources available in the project
* refactor: Update Plan struct's Backend field to be a pointer
* fix: Add some refactorings or protections against nil pointers when accessing the Backend field on a Plan struct
* refactor: Update Plan struct's StateStore field to be a pointer, update a nil check in calling code.
* Add a generic method for loading an operations backend in non-init commands
* Refactor commands to use new prepareBackend method: group 1
* Refactor commands to use new prepareBackend method: group 2, where config parsing needs to be explicitly added
* Refactor commands to use new prepareBackend method: group 3, where we can use already parsed config
* Additional, more nested, places where logic for accessing backends needs to be refactored
* Remove duplicated comment
* Add test coverage of `(m *Meta) prepareBackend()`
* Add TODO related to using plans for backend/state_store config in apply commands
* Add `testStateStoreMockWithChunkNegotiation` test helper
* Add assertions to tests about the backend (remote-state, local, etc) in use within operations backend
* Stop prepareBackend taking locks as argument
* Code comment in prepareBackend
* Replace c.Meta.prepareBackend with c.prepareBackend
* Change `c.Meta.loadSingleModule` to `c.loadSingleModule`
* Rename (Meta).prepareBackend to (Meta).backend, update godoc comment to make relationship to (Meta).Backend more obvious.
* Revert change from config.Module to config.Root.Module
* Update `(m *Meta) backend` method to parse config itself, and also to adhere to calling code's viewtype instructions
* Update all tests and calling code following previous commit
* Change how an operations backend is obtained by autocomplete code
* Update autocomplete to return nil if no workspace names are returned from the backend
* Add test coverage for autocompleting workspace names when using a pluggable state store
* Fix output command: pass view type data to new `backend` method
* Fix in plan command: pass correct view type to `backend` method
* Fix `providers schema` command to use correct viewtype when preparing a backend
* Update comments to distinguish between operations and state-storage backends more clearly
* Rename `BackendOpts` field `Config` to more specific `BackendConfig`
* include help entry for -replace
* add to website docs as well
* Update internal/command/apply.go
Co-authored-by: Daniel Banck <dbanck@users.noreply.github.com>
* Update website/docs/cli/commands/apply.mdx
Co-authored-by: Daniel Banck <dbanck@users.noreply.github.com>
---------
Co-authored-by: Daniel Banck <dbanck@users.noreply.github.com>
* show deprecation warning if -state is used with plan, apply, refresh
* show deprecation warning if -state is used with plan, apply, refresh
* updated -state flag check condition
* added better diagnostic details view
* resolved failed test
* updated the content of the -state flag warning
To support ephemeral values we need a more complicated set of rules about
what input variables can and must be set when applying a saved plan. The
command itself does not have enough information to implement those rules
itself, so we'll let them pass through and check this in the local
backend's apply phase instead.
The local backend's apply phase already had basic support for dealing with
apply-time variable values, but it'll now also be responsible for
rejecting attempts to set variables when the experiment isn't enabled,
to keep all of this logic in roughly the same place.
Now that deferred actions is a top-level per-plan option, here's a lil
command-line option for turning it on!
- It's called `-allow-deferral`.
- It is ONLY legal to use if this is an experimental build, i.e. compiled with
`go build -ldflags "-w -s -X 'main.experimentsAllowed=yes'"` or the like.
Implementation notes: The design constraint here was, "please avoid having to
change the build infrastructure or alter the function of every args parsing
function". So, since `extendedFlagSet()` doesn't have access to the var that
says we're in an experimental build, I had to move the validation for that
particular flag to later in the process, when we're building the operation
request.
We previously had all of the types and helpers for all kinds of backends
together in package backend. That kept things relatively simple, but it
also meant that the majority of backends that only deal with remote state
storage ended up still indirectly depending on the entire Terraform modules
runtime, configuration loader, etc, etc, which brings into scope a bunch
of external dependencies that the remote state backends don't really need.
Since backends that support operations are a rare exception, we'll move the
types and helpers for those into a separate package "backendrun", and
then the main package backend can have a much more modest set of types and,
more importantly, a modest set of dependencies on other packages in this
codebase.
This is part of an ongoing effort to reduce the exposure of Terraform Core
and CLI code to the remote backends and vice-versa, so that in the long
run we can more often treat them as separate for dependency maintenance
purposes.
For a very long time we've had an annoying discrepancy between the
in-memory state model and our state snapshot format where the in-memory
format stores output values for all modules whereas the snapshot format
only tracks the root module output values because those are all we
actually need to preserve between runs.
That design wart was a result of us using the state both as an internal
and an external artifact, due to having nowhere else to store the
transient values of non-root module output values while Terraform Core
does its work.
We now have namedvals.State to internally track all of the throwaway
results from named values that don't need to persist between runs, so now
we'll use that for our internal work instead and reserve the states.State
model only for the data that we will preserve between runs in state
snapshots.
The namedvals internal model isn't really designed to support enumerating
all of the output values for a particular module call, but our expression
evaluator currently depends on being able to do that and so we have a
temporary inefficient implementation of that which just scans the entire
table of values as a stopgap just to avoid this commit growing even larger
than it already is. In a future commit we'll rework the evaluator to
support the PartialEval mode and at the same time move the responsiblity
for enumerating all of the output values into the evaluator itself, since
it should be able to determine what it's expecting by analyzing the
configuration rather than just by trusting that earlier evaluation has
completed correctly.
Because our legacy state string serialization previously included output
values for all modules, some of our context tests were accidentally
depending on the implementation detail of how those got stored internally.
Those tests are updated here to test only the data that is a real part
of Terraform Core's result, by ensuring that the relevant data appears
somewhere either in a root output value or in a resource attribute.
It displays a run header with link to web UI, like starting a new plan does, then confirms the run
and streams the apply logs. If you can't apply the run (it's from a different workspace, is in an
unconfirmable state, etc. etc.), it displays an error instead.
Notable points along the way:
* Implement `WrappedPlanFile` sum type, and update planfile consumers to use it instead of a plain `planfile.Reader`.
* Enable applying a saved cloud plan
* Update TFC mocks — add org name to workspace, and minimal support for includes on MockRuns.ReadWithOptions.
* Add viewType to Meta object and use it at the call sites
* Assign viewType passed from flags to state-locking cli commands
* Remove temp files
* Set correct mode for statelocker depending on json flag passed to commands
* Add StateLocker interface conformation check for StateLockerJSON
* Remove empty line at end of comment
* Pass correct ViewType to StateLocker from Backend call chain
* Pass viewType to backend migration and initialization functions
* Remove json processing info in process comment
* Restore documentation style of backendMigrateOpts
When using the Terraform Cloud integration - like the 'remote'
backend - resource count output should be suppressed if those counts are
being rendered remotely. This generalizes this to the shared
BackendWithRemoteTerraformVersion interface.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
