* refactor: Check that the state storage provider is present when beginning to initialise a state store for use in a non-init command. Ensure reattached providers can be used.
Previously we passed all required providers into backend options to be used within `stateStoreConfig`, which is invoked via (Meta).Backend. The new approach enforces that the provider is present while assembling the backend options passed to (Meta).Backend from (Meta).backend, which is non-init specific. As this code is defending against users running non-init commands before an init, this place feels appropriate and isn't able to impact the init command.
* fix: Reattached PSS providers should return early when checking locks, and an empty locks file is only bad if there isn't a reattached PSS provider
* test: Assert that running init with reattached PSS provider is ok, via an E2E test that uses the reattach feature.
* fix: Allow builtin or reattached providers to be used for state stores when generating a plan file
* test: Expand E2E test to show using a reattached provider can be used for a workflow of init, plan with -out, and apply.
* chore: Replace 'io/ioutil' and format code in unmanaged e2e tests
* backend/inmem: Make it easier to use the backend in tests
* cloud: Make the cloud backend testable
* command/views: Introduce migration UI messages
* command/init: Add 3 tests for migrations from a backend to PSS
- TestInit_backend_to_stateStore_singleWorkspace
- TestInit_backend_to_stateStore_multipleWorkspaces
- TestInit_cloud_to_stateStore
* command/init: Implement migration from a backend to PSS
* address PR feedback
* remove local state after migration
* fix: Fail apply command if the plan file was generated for a workspace that isn't the selected workspace.
* Add change file
* test: Update test helper to include Workspace name in plan representation
* fix: Make error message more generic, so is applicable to backend and cloud blocks.
* fix: Make error message specific to backend or cloud block
* test: Add separate tests for backend/cloud usage
* test: Update remaining tests to include a value for Workspace in mocked plans
* Apply suggestions from code review
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* fix: Panic when a plan file has missing workspace data
* test: Update test to match changes in error text
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* feat: Allow reading state store configuration from a planfile and using it to prepare a Local backend that uses the state store
* test: Assert that we can get and use state store configuration from a plan file
* test: Add integration test showing that an apply command can use a plan file to configure and use a state store
* test: Add E2E test showing pluggable state storage being used with the full init-plan-apply workflow
* feat: A plan file will report the state storage provider among its required providers, if PSS is in use.
See the code comment added in this commit. This addition does not impact an apply command as the missing provider will be detected before this code is executed. However I'm making this change so that the method is still accurate is being able to return a complete list of providers needed by the plan.
* fix: Include error messages when there is a problem parsing provider or state store config when getting a backend from a planfile
* feat: Add trace logs to BackendForLocalPlan indicating when the provider is launched and the state store is configured
* chore: Small grammar change in error diagnostic
* refactor: Remove suggestions when the plan's state store doesn't match the implementations in the provider
* test: Add test coverage of what happens when the contents of a plan file using PSS doesn't match the resources available in the project
* command/meta: Enable migration from PSS to a backend
* Address PR feedback
* Update internal/command/meta_backend.go
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
* meta_backend: Rename stateStore_c_S to stateStore_to_backend
---------
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
* refactor: Rename Meta's backendState field to backendConfigState
This helps with navigating ambiguity around the word backend. The new name should indicate that the value represents a `backend` block, not a more general interpretation of what a backend is.
* fix: Only set backendConfigState to synthetic object if it's nil due to a lack of data. Don't change it if pluggable state storage is in use.
* feat: Enable recording a state store's details in an Operation, and using that data when creating a plan file.
* fix: Include provider config when writing a plan file using pluggable state storage
* fix: Having `backendConfigState` be nil may be valid, but it definitely isn't valid for `stateStoreConfigState` to be nil
When backendConfigState is nil it means that an implied local backend is in use, i.e. there is no backend block in the config.
* test: Add integration test showing that a plan command creates a plan file with the expected state_store configuration data
* refactor: Apply suggestion from @radeksimko
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* fix: Allow panics to occur if an unexpected implementation of `backend.Backend` is encountered when managing a state store
* docs: Add code comment explaining the current situation with passing backend config state to downstream logic.
In future this should be simplified, either via refactoring or changes affecting the implied local backend
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* refactor: Move chunk size limit constants to new `chunks` package
* refactor: Make `NewPluggable` return a `Pluggable` concrete type, instead of an instance of the `backend.Backend` interface.
* refactor: Configure state stores through the backend.Backend interface, instead of directly using methods related to RPCs. This requires changing where we call `SetStateStoreChunkSize`.
* docs: Add godocs comment to `StateStoreChunkSizeSetter` interface
To summarize, we don't really need the `SetStateStoreChunkSize` method, and instead methods like `(*GRPCProvider).ConfigureStateStore` in the `plugin6` package can directly inspect the negotiation process that passes through that code and pull out the chunk size. However that means that that code would also need to implement validation. And that's just `(*GRPCProvider).ConfigureStateStore`; what about all the test mocks that are used in different types of test? They would all need to be implemented similarly to GRPCProvider to be good, useful mocks, and then a lot of things that fulfil the provider.Interface interface are coupled to each other. Instead, it's easier to have validation in the `grpcClient` struct's methods in the `remote` package, as that code is common to all scenarios. That code needs a method to 'reach into' the provider.Interface value, so we use the `SetStateStoreChunkSize` method.
* chore: Make it clearer that the v6 GRPCProvider implements `SetStateStoreChunkSize`
* fix: Remove unnecessary assignment of chunk size
I'm surprised that removing this doesn't break E2E tests of PSS that use grpcwrap, but I think there's `plugin6` code that runs in that situation, so maybe chunking is handled elsewhere.
* chore: Add panic to try detect unexpected cases when setting chunk size.
* feat: Add `providers.StateStoreChunkSizeSetter` implementation to provider-simple-v6
* docs: Update code comments for NewPluggable describing its intended use
* fix: Make all init backend messages JSON-compatible
Fixes#37911 by converting all backend configuration messages to use the
View abstraction, which properly formats output as JSON when the -json flag
is used. Previously, certain messages were output directly using the legacy
Ui abstraction, bypassing JSON formatting.
Changes:
- Added 11 new message codes to init.go MessageRegistry with both human
and JSON formatted values
- Replaced 11 direct m.Ui.Output() calls in meta_backend.go with View
abstraction calls
- Fixed output in backend_C_r_s(), backend_C_r_S_changed(), backend_c_r_S(),
and stateStore_c_S() functions
All init tests pass successfully.
* docs: Add changelog entry for init JSON backend messages fix
* cleanup: Remove unused backend output constants
The following constants were moved to init.go as message registry entries
and are no longer used in meta_backend.go:
- outputBackendMigrateChange
- outputBackendMigrateLocal
- outputStateStoreMigrateLocal
- outputBackendReconfigure
- successBackendUnset
- successBackendSet
* feat: Implement `inmem` state store in provider-simple-v6
* feat: Add filesystem state store `fs` in provider-simple-v6, no locking implemented
* refactor: Move PSS chunking-related constants into the `pluggable` package, so they can be reused.
* feat: Implement PSS-related methods in grpcwrap package
* test: Add E2E test checking an init and apply (no plan) workflow is usable with both PSS implementations
* fix: Ensure state stores are configured with a suggested chunk size from Core
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Fix panic hashing backend config when a builtin or reattached provider is used for PSS
* Add test showing a nil *version.Version is tolerated by the Hash method, refactor existing test
* Protect against provider version data missing when using an un-reattached, un-builtin provider. Add test coverage.
* Make reporting errors from reattach.IsProviderReattached more consistent
* Run `make syncdeps`
* Update tests to show when nil provider data is tolerated
* Add a generic method for loading an operations backend in non-init commands
* Refactor commands to use new prepareBackend method: group 1
* Refactor commands to use new prepareBackend method: group 2, where config parsing needs to be explicitly added
* Refactor commands to use new prepareBackend method: group 3, where we can use already parsed config
* Additional, more nested, places where logic for accessing backends needs to be refactored
* Remove duplicated comment
* Add test coverage of `(m *Meta) prepareBackend()`
* Add TODO related to using plans for backend/state_store config in apply commands
* Add `testStateStoreMockWithChunkNegotiation` test helper
* Add assertions to tests about the backend (remote-state, local, etc) in use within operations backend
* Stop prepareBackend taking locks as argument
* Code comment in prepareBackend
* Replace c.Meta.prepareBackend with c.prepareBackend
* Change `c.Meta.loadSingleModule` to `c.loadSingleModule`
* Rename (Meta).prepareBackend to (Meta).backend, update godoc comment to make relationship to (Meta).Backend more obvious.
* Revert change from config.Module to config.Root.Module
* Update `(m *Meta) backend` method to parse config itself, and also to adhere to calling code's viewtype instructions
* Update all tests and calling code following previous commit
* Change how an operations backend is obtained by autocomplete code
* Update autocomplete to return nil if no workspace names are returned from the backend
* Add test coverage for autocompleting workspace names when using a pluggable state store
* Fix output command: pass view type data to new `backend` method
* Fix in plan command: pass correct view type to `backend` method
* Fix `providers schema` command to use correct viewtype when preparing a backend
* Pull determining of PSS provider's version from current locks into a separate method
* Add code for identifying when config and provider version match existing backend state (i.e. no changes)
* Update test - locks are now needed before it hits expected error diag return
* Add test showing successful init when no config changes are detected.
* Update `getStateStorageProviderVersion` to return nil versions for builtin and re-attached providers.
This makes comparison easier when determining if config has changed since last init.
* Add test coverage for `getStateStorageProviderVersion`
* Move testing fixtures around, preparing for different types of changed state_store config changes being tested
* Add test showing that changing the state_store config is detected as a change, but handling this scenario isn't implemented yet
* Update hashes in test fixture backend state file to be accurate
Previously dummy values were fine, but as tests using hashes to identify changes these values need to be accurate!
* Update existing test cases so that Terraform uses the same test provider version as described in the backend state file fixture for the test.
* Add test showing that changing the PSS provider's config is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that swapping to a different state storage implementation in the same provider is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that changing the provider used for PSS is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that upgrading a provider is detected as a change, but handling this scenario isn't implemented yet
* Update test to use v1.2.3 for consistency with other tests
Just to avoid any confusion if copy-pasting happens in future.
* More corrections to existing test fixtures - unset config should be null, and replace dummy hash values with correct values.
* Fix test for using -reconfigure with state_store; the default workspace would already exist in this scenario
* Update TestInit_stateStore_configUnchanged to assert that init was a no-op for backend state
* Remove unused fixture
* Remove test that's replaced by new tests in command/init_test.go
* Replace old references to deleted "state-store-changed" test fixture & update test to not expect a value for region attr in provider config
* Make test fixture coupling a little more understandable
* Refactor detection of no need to migrate into a function
* Add TODO about more involved provider version change tests
We will allow downgrades to succeed as long as the schema version number is unchanged
* Update (configs.StateStore)Hash method to return a single hash that's impacted by: state store config, provider config, state store type, provider source
* Update calling code and test helper code to reflect that the nested provider block no longer has its own hash
* Remove test; there is now a single hash that SHOULD be affected by the provider block!
* Also use provider name, from config, in hash
* Update tests to reflect changes in how hashes are made
* Remove unused `stateStoreConfigNeedsMigration` function
* Remove duplicate isProviderReattached function.
* Fixes to affected tests
* Allow provider version to impact the state storage hash, update impacted tests and test fixtures
* Update tests that now require locks data to be present in test setup
* Update comment for accuracy
* Fixes to other test fixtures - remove excess hash field, set hash to 0 to indicate they're not set accurately.
* Make upgrade test actually use upgrade code path
* Add lock files to test fixture directories that represent a project that's had a successful prior init using PSS
* Minor fixes in diagnostics
This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.
* Rename test to make it specific to use of backend block in config
* Update initBackend to accept whole initArgs collection
* Only process --backend-config data, when setting up a `backend`, if that data isn't empty
* Simplify how mock provider factories are made in tests
* Update mock provider's default logic to track and manage existing workspaces
* Add `ProviderSchema` method to `Pluggable` structs. This allows calling code to access the provider schema when using provider configuration data.
* Add function for converting a providerreqs.Version to a hashicorp/go-version Version.
This is needed for using locks when creating the backend state file.
* Implement initial version of init new working directories using `stateStore_C_s`. Default to creating the default workspace if no workspaces exist.
* Update test fixtures to match the hashicorp/test mock provider used in PSS tests
* Allow tests to obtain locks that include `testingOverrides` providers.
The `testingOverrides` field will only be set in tests, so this should not impact end users.
* Add tests showing TF can initialize a working directory for the first time (and do the same when forced by -reconfigure flag). Remove replaced tests.
* Add -create-default-workspace flag, to be used to disable creating the default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests.
* Allow reattached providers to be used during init for PSS
* Rename variable to `backendHash` so relation to `backend` is clearer
* Allow `(m *Meta) Backend` to return warning diagnostics
* Protect against nil testingOverrides in providerFactoriesFromLocks
* Add test case seeing what happens if default workspace selected, doesn't exist, but other workspaces do exist.
The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`.
* Address code consistency check failure on PR
* Refactor use of mock in test that's experiencing EOF error...
* Remove test that requires test to supply input for user prompt
This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input.
* Allow -create-default-workspace to be used regardless of whether input is enabled or disabled
* Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable
* Responses to feedback, including making testStdinPipe helper log details of errors copying data to stdin.
Note: We cannot call t.Fatal from a non-test goroutine.
* Use Errorf instead
* Allow backend state files to not include version data when a builtin or reattached provider is in use.
* Add clarifying comment about re-attached providers when finding the matching entry in required_providers
* Report that the default workspace was created to the view
* Refactor: use error comparison via `errors.Is` to identify when no workspaces exist.
* Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseInit func.
* Validate that PSS-related flags can only be used when experiments are enabled, enforce coupling of PSS-related flags when in use.
* Slight rewording of output message about default workspace
* Update test to assert new output about default workspace
* Replace `errBackendLocalRead` constant with custom error of same name
* Replace `errBackendNoExistingWorkspaces` constant with custom error of same name
* Replace `errBackendMigrateLocalDelete` constant with custom error of same name
* Replace `errBackendNewUnknown` constant with custom error of same name
* Replace `errBackendSavedUnknown` constant with custom error of same name
* Replace `errBackendClearSaved` constant with custom error of same name
* Replace `errBackendInit` constant with custom error of same name
* Replace `errBackendInitCloud` constant with custom error of same name
* Replace `errBackendWriteSaved` constant with custom error of same name
* Move `migrateOrReconfigDiag` next to custom errors
* Replace `errBackendInit` custom error with `func errBackendInitDiag` for creating diagnostics directly
* Replace `errBackendInitCloud` custom error with `func errBackendInitCloudDiag` for creating diagnostics directly
* Replace `errBackendWriteSaved` custom error with `func errBackendWriteSavedDiag` for creating diagnostics directly
* Add new usage of errBackendWriteSavedDiag; this is another place where failing to write to backend state files happens
* Add godoc comments
* Ensure errors aren't repeated in terminal output, and ensure warnings are not lost.
* Add SetStateStoreChunkSize to the mock provider for tests
* Implement configurable state chunk size
* Add chunk size negotiation to `savedStateStore`, update happy path test to assert it's set
* Update `savedStateStore` to return diagnostic if a nil factory is passed in, add unhappy path tests
* Add test coverage for stateStoreConfig, including use of config overrides with state stores
* Add godoc comment for backendInitFromConfig
* Avoid nil error in stateStoreConfig when there's no provider factory
* Add stateStoreInitFromConfig method and test coverage
* Refactor stateStoreInitFromConfig to accept factory directly, instead of accepting all BackendOpts
This is to avoid confusion, e.g. unnecessary duplication of config being passed in.
* Implement configurable state chunk size
* Remove TODO which has since been addressed
* Update happy path test for `stateStoreInitFromConfig` to check that chunk size is negotiated
* Update chunk size negotiation code in `stateStoreInitFromConfig` to error if the provider doesn't return a chunk size > 0
* Fix test error message
---------
Co-authored-by: Radek Simko <radek.simko@gmail.com>
* Add SetStateStoreChunkSize to the mock provider for tests
* Implement configurable state chunk size
* Add chunk size negotiation to `savedStateStore`, update happy path test to assert it's set
* Update `savedStateStore` to return diagnostic if a nil factory is passed in, add unhappy path tests
* Fix test error message
* Apply suggestions from code review
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Fix rename
---------
Co-authored-by: Radek Simko <radek.simko@gmail.com>
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Add test coverage for Meta's `savedBackend` method
* Add new Meta `savedStateStore` method and test coverage
* Streamline test - remove unneeded assertions and update comments
* Remove marks from config before configuring the provider
* Remove marks from config before configuring the state store
* Add test case for savedStateStore to assert marks aren't passed
* Fix call to ConfigureStateStore
* Show that tests pass despite not trying to remove marks
* Allow Config methods to add marks when reading pluggable state store config from the backend state file
* This code is now necessary to let the tests pass
* Stop adding marks to PSS-related config when it's parsed from the backend state file
* Stop removing marks that aren't there
* Remove unnecessary test related to marks
* Add method to allow accessing factories from locks that are in memory
* Create new getStateStoreProviderFactory method for accessing a factory from config
* Update getStateStoreProviderFactory to use in memory locks instead of reading them from the deps lock file. Update calling code to accommodate this.
* Add tests for getStateStoreProviderFactory, improve errors returned from method
* Update test following schema change in simple providers
* Move test case into e2etest package, so we protect against environments where building binaries isn't possible
* Fix issues with running test in e2etest package
* Update code comments
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Update call to StateMgr method to use diagnostics more explicitly
* Update calling code in remote backend tests
* Update calling code in non-test code to handle diags conventionally.
* Fix test failure messages from using wrong variable
* Additional refactoring
* Update Backend interface to use diagnostics for `DeleteWorkspace` and `Workspaces` methods
* Update testing helpers in backend package
* Update pluggable package to match interface changes
* Update builtin terraform provider package to match interface changes
* Update code in Meta-related files in command package; update error check logic
* Update workspace commands; update error check logic
* Update Local backend package to match interface changes
* Update Cloud backend to match interface changes
* Update Remote backend to match interface changes
* Update cos remote-state backend to match interface changes
* Update inmem remote-state backend to match interface changes
* Update pg remote-state backend to match interface changes
* Update http remote-state backend to match interface changes
* Update consul remote-state backend to match interface changes
* Update oss remote-state backend to match interface changes
* Update kubernetes remote-state backend to match interface changes
* Update gcs remote-state backend to match interface changes
* Update s3 remote-state backend to match interface changes
* Update oci remote-state backend to match interface changes
* Update azure remote-state backend to match interface changes
* Allow warnings from Workspaces and DeleteWorkspace methods to be logged or returned to calling code
* Fix defect in inmem backend test
* Change how warnings from workspace-related methods are output to view/ui
* Store the FQN of the provider used in PSS in representations of the parsed config.
This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.
* Update affected tests, improve error diagnostic
* Begin enabling method to return a backend.Backend made using state_store config.
State store config can now be received via BackendOpts and there is rough validation of whether the config makes sense (does the provider offer a store with the given name?).
* Update code's logic to include possibility of a state store being in use
At this point there are no cases that actually handle the state store scenarios though!
* Add empty cases for handle all broad init scenarios involving PSS
* Update default case's error to report state store variables
* Improve how we resolve the builtin terraform provider's address
* Add test that hits the code path for adding a state store to a new (or implied local) project
* Add test for use of `-reconfigure` flag; show that it hits the code path for adding a state store for the first time
* Add test that hits the code path for removing use of a state store, migrating to (implied) local backend
* Add test that hits the code path for changing a state store's configuration
* Update existing test names to be backend-specific
* Add tests that hits the code path for migrating between PSS and backends
* Consolidate PSS-related tests at end of the file
* Fix log text
* Add test showing that using variables is disallowed with state_store and nested provider blocks
* Update test name
* Fix test cases
* Add TODOs so we remember to remove experiments from tests
* Update state store-related tests to use t.Cleanup
* Remove use of `testChdir`
* Update comments to distinguish between operations and state-storage backends more clearly
* Rename `BackendOpts` field `Config` to more specific `BackendConfig`
* Add test coverage for Meta's determineInitReason method, fix error in diagnostic when migrating away from HCP Terraform
* Add godoc comment to determineInitReason method
* Implement `ForPlan` method on `StateStoreConfigState`, add `Planner` interface
* Rename `ForPlan` method to `Plan`
* Allow plan files to contain information about state stores
* Add code needed for representing a state store in the Plan struct, which is used when handling plan files
* Add ability to read/write either a backend or state store's data in a plan file. Add some test coverage.
* Update plan's `ProviderAddrs` method to include the provider used for PSS, if present
* Split interfaces
* Apply feedback from code review
* Refactor `SetVersion` to use appropriate constructor
* Split `ProviderAddrs` method test into two
* Fix method call after rename
* Fix test
* Remove change to `(p *Plan) ProviderAddrs()`
We may re-add this when we implement PSS for use during apply commands with plan files
* Remove changes to test, now that the plan doesn't report the provider used for PSS anymore.
* Split code for backend state file vs backend state
* Rename BackendState to BackendConfigState
* Spelling error
* Add `StateStorageConfigState` struct as new implementation of new `ConfigState[T any]` interface.
* Split tests for backend state file vs backend config state structs
* Rename StateStorageConfigState to StateStoreConfigState
* Clarify test name, add comments
* Add tests for StateStoreConfigState methods
* Add test showing state_store in JSON is parsed correctly
* Add detection of malformed backend state files that contain both backend and state_store fields
* Add validation that stops a backend state file being written if it will contain state for both backend and state_store blocks
* Rename `state_storage` to `state_store`
* Rename `state_storage` to `state_store` in filenames
* Move`ConfigState` to its own file
* Fix test name, remove whitespace
* Update `StateStoreConfigState` comment using review suggestion
* Update error message to no longer allude to the environment TF is being run in
* Update the state_store state to use `version.Version` and an adapted version of `tfaddr.Provider` for marshalling version and source data
* Update test helper so it doesn't accidentally supply validation in tests
* Add protection against saving an empty backend state file
* Remove direct testing of (s *Source) MarshalText() and UnmarshalText() methods
* Add Validate method to StateStoreConfigState, use in backend state encoding logic
* Refactor to use new features in registry dependency
* Fix copy-pasta
* Update some comments clarifying backend-related interfaces and "enhanced" versus "operations"
* Fix more comments that refer to types and interfaces that have moved into the backendrun package
When key-value tags are enabled and used in the workspace, users may
define the tags attribute as a map of strings in the cloud block in order
to more precicely match workspaces using those tags.
Some updates to staticcheck were catching more errors. Most of these
were format functions without a format string, but rather than adding
more exceptions I just fixed them all. This did also catch some
incorrectly wrapped errors.
We previously had all of the types and helpers for all kinds of backends
together in package backend. That kept things relatively simple, but it
also meant that the majority of backends that only deal with remote state
storage ended up still indirectly depending on the entire Terraform modules
runtime, configuration loader, etc, etc, which brings into scope a bunch
of external dependencies that the remote state backends don't really need.
Since backends that support operations are a rare exception, we'll move the
types and helpers for those into a separate package "backendrun", and
then the main package backend can have a much more modest set of types and,
more importantly, a modest set of dependencies on other packages in this
codebase.
This is part of an ongoing effort to reduce the exposure of Terraform Core
and CLI code to the remote backends and vice-versa, so that in the long
run we can more often treat them as separate for dependency maintenance
purposes.
Historically (before there were "backends") Terraform had a single state
file format used both for real state snapshots and for tracking where
remote state was stored.
Terraform v0.12 caused these two to be split because we adopted state
snapshot version 4 for real snapshots but retained a subset of version 3
for tracking the remote backend configuration in the local working
directory.
Unfortunately we previously kept that working by retaining a snapshot of
the entire Terraform v0.11 "terraform" package as legacy/terraform, which
happened to still be around because we also needed to retain a copy of
the entire legacy SDK to keep the remote state backends working.
This now hoists just the tiny slice of legacy terraform package
functionality needed to implement the "backend state" file format into
package workdir. This package is a good home for it because it's part of
the working directory state. Ideally it would be accessed through methods
of the workdir.Dir type, but that's too disruptive a refactor to combine
into this and so that'll need to wait for another day; for now we'll keep
the existing callers doing their access through our "clistate" package
that is itself a forked snapshot of what statemgr.Filesystem used to be
in Terraform v0.11.
This removes all but one of the uses of "legacy/terraform" aside from calls
in the other packages under "legacy". We'll clean up the last one in a
later commit, because it's not related to the backend state file format.
It displays a run header with link to web UI, like starting a new plan does, then confirms the run
and streams the apply logs. If you can't apply the run (it's from a different workspace, is in an
unconfirmable state, etc. etc.), it displays an error instead.
Notable points along the way:
* Implement `WrappedPlanFile` sum type, and update planfile consumers to use it instead of a plain `planfile.Reader`.
* Enable applying a saved cloud plan
* Update TFC mocks — add org name to workspace, and minimal support for includes on MockRuns.ReadWithOptions.
Previously, remote and cloud backends would automatically alias localterraform.com as the configured hostname during configuration. This turned out to be an issue with how backends could potentially be used within the builtin terraform_remote_state data source. Those data sources each configure the same service discovery with different targets for localterraform.com, and do so simultaneously, creating an occasional concurrent map read & write panic when multiple data sources are defined.
localterraform.com is obviously not useful for every backend configuration. Therefore, I relocated the alias configuration to the callers, so they may specify when to use it. The modified design adds a new method to backend.Enhanced to allow configurators to ask which aliases should be defined.
Several times over the years we've considered adding tracing
instrumentation to Terraform, since even when running in isolation as a
CLI program it has a "distributed system-like" structure, with lots of
concurrent internal work and also some work delegated to provider plugins
that are essentially temporarily-running microservices.
However, it's always felt a bit overwhelming to do it because much of
Terraform predates the Go context.Context idiom and so it's tough to get
a clean chain of context.Context values all the way down the stack without
disturbing a lot of existing APIs.
This commit aims to just get that process started by establishing how a
context can propagate from "package main" into the command package,
focusing initially on "terraform init" and some other commands that share
some underlying functions with that command.
OpenTelemetry has emerged as a de-facto industry standard and so this uses
its API directly, without any attempt to hide it behind an abstraction.
The OpenTelemetry API is itself already an adapter layer, so we should be
able to swap in any backend that uses comparable concepts. For now we just
discard the tracing reports by default, and allow users to opt in to
delivering traces over OTLP by setting an environment variable when
running Terraform (the environment variable was established in an earlier
commit, so this commit builds on that.)
When tracing collection is enabled, every Terraform CLI run will generate
at least one overall span representing the command that was run. Some
commands might also create child spans, but most currently do not.
