* refactor: Check that the state storage provider is present when beginning to initialise a state store for use in a non-init command. Ensure reattached providers can be used.
Previously we passed all required providers into backend options to be used within `stateStoreConfig`, which is invoked via (Meta).Backend. The new approach enforces that the provider is present while assembling the backend options passed to (Meta).Backend from (Meta).backend, which is non-init specific. As this code is defending against users running non-init commands before an init, this place feels appropriate and isn't able to impact the init command.
* fix: Reattached PSS providers should return early when checking locks, and an empty locks file is only bad if there isn't a reattached PSS provider
* test: Assert that running init with reattached PSS provider is ok, via an E2E test that uses the reattach feature.
* fix: Allow builtin or reattached providers to be used for state stores when generating a plan file
* test: Expand E2E test to show using a reattached provider can be used for a workflow of init, plan with -out, and apply.
* chore: Replace 'io/ioutil' and format code in unmanaged e2e tests
* refactor: Rename Meta's backendState field to backendConfigState
This helps with navigating ambiguity around the word backend. The new name should indicate that the value represents a `backend` block, not a more general interpretation of what a backend is.
* fix: Only set backendConfigState to synthetic object if it's nil due to a lack of data. Don't change it if pluggable state storage is in use.
* feat: Enable recording a state store's details in an Operation, and using that data when creating a plan file.
* fix: Include provider config when writing a plan file using pluggable state storage
* fix: Having `backendConfigState` be nil may be valid, but it definitely isn't valid for `stateStoreConfigState` to be nil
When backendConfigState is nil it means that an implied local backend is in use, i.e. there is no backend block in the config.
* test: Add integration test showing that a plan command creates a plan file with the expected state_store configuration data
* refactor: Apply suggestion from @radeksimko
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* fix: Allow panics to occur if an unexpected implementation of `backend.Backend` is encountered when managing a state store
* docs: Add code comment explaining the current situation with passing backend config state to downstream logic.
In future this should be simplified, either via refactoring or changes affecting the implied local backend
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Fix panic hashing backend config when a builtin or reattached provider is used for PSS
* Add test showing a nil *version.Version is tolerated by the Hash method, refactor existing test
* Protect against provider version data missing when using an un-reattached, un-builtin provider. Add test coverage.
* Make reporting errors from reattach.IsProviderReattached more consistent
* Run `make syncdeps`
* Update tests to show when nil provider data is tolerated
* Pull determining of PSS provider's version from current locks into a separate method
* Add code for identifying when config and provider version match existing backend state (i.e. no changes)
* Update test - locks are now needed before it hits expected error diag return
* Add test showing successful init when no config changes are detected.
* Update `getStateStorageProviderVersion` to return nil versions for builtin and re-attached providers.
This makes comparison easier when determining if config has changed since last init.
* Add test coverage for `getStateStorageProviderVersion`
* Move testing fixtures around, preparing for different types of changed state_store config changes being tested
* Add test showing that changing the state_store config is detected as a change, but handling this scenario isn't implemented yet
* Update hashes in test fixture backend state file to be accurate
Previously dummy values were fine, but as tests using hashes to identify changes these values need to be accurate!
* Update existing test cases so that Terraform uses the same test provider version as described in the backend state file fixture for the test.
* Add test showing that changing the PSS provider's config is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that swapping to a different state storage implementation in the same provider is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that changing the provider used for PSS is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that upgrading a provider is detected as a change, but handling this scenario isn't implemented yet
* Update test to use v1.2.3 for consistency with other tests
Just to avoid any confusion if copy-pasting happens in future.
* More corrections to existing test fixtures - unset config should be null, and replace dummy hash values with correct values.
* Fix test for using -reconfigure with state_store; the default workspace would already exist in this scenario
* Update TestInit_stateStore_configUnchanged to assert that init was a no-op for backend state
* Remove unused fixture
* Remove test that's replaced by new tests in command/init_test.go
* Replace old references to deleted "state-store-changed" test fixture & update test to not expect a value for region attr in provider config
* Make test fixture coupling a little more understandable
* Refactor detection of no need to migrate into a function
* Add TODO about more involved provider version change tests
We will allow downgrades to succeed as long as the schema version number is unchanged
* Update (configs.StateStore)Hash method to return a single hash that's impacted by: state store config, provider config, state store type, provider source
* Update calling code and test helper code to reflect that the nested provider block no longer has its own hash
* Remove test; there is now a single hash that SHOULD be affected by the provider block!
* Also use provider name, from config, in hash
* Update tests to reflect changes in how hashes are made
* Remove unused `stateStoreConfigNeedsMigration` function
* Remove duplicate isProviderReattached function.
* Fixes to affected tests
* Allow provider version to impact the state storage hash, update impacted tests and test fixtures
* Update tests that now require locks data to be present in test setup
* Update comment for accuracy
* Fixes to other test fixtures - remove excess hash field, set hash to 0 to indicate they're not set accurately.
* Make upgrade test actually use upgrade code path
* Add lock files to test fixture directories that represent a project that's had a successful prior init using PSS
* Minor fixes in diagnostics
This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.
* Rename test to make it specific to use of backend block in config
* Update initBackend to accept whole initArgs collection
* Only process --backend-config data, when setting up a `backend`, if that data isn't empty
* Simplify how mock provider factories are made in tests
* Update mock provider's default logic to track and manage existing workspaces
* Add `ProviderSchema` method to `Pluggable` structs. This allows calling code to access the provider schema when using provider configuration data.
* Add function for converting a providerreqs.Version to a hashicorp/go-version Version.
This is needed for using locks when creating the backend state file.
* Implement initial version of init new working directories using `stateStore_C_s`. Default to creating the default workspace if no workspaces exist.
* Update test fixtures to match the hashicorp/test mock provider used in PSS tests
* Allow tests to obtain locks that include `testingOverrides` providers.
The `testingOverrides` field will only be set in tests, so this should not impact end users.
* Add tests showing TF can initialize a working directory for the first time (and do the same when forced by -reconfigure flag). Remove replaced tests.
* Add -create-default-workspace flag, to be used to disable creating the default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests.
* Allow reattached providers to be used during init for PSS
* Rename variable to `backendHash` so relation to `backend` is clearer
* Allow `(m *Meta) Backend` to return warning diagnostics
* Protect against nil testingOverrides in providerFactoriesFromLocks
* Add test case seeing what happens if default workspace selected, doesn't exist, but other workspaces do exist.
The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`.
* Address code consistency check failure on PR
* Refactor use of mock in test that's experiencing EOF error...
* Remove test that requires test to supply input for user prompt
This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input.
* Allow -create-default-workspace to be used regardless of whether input is enabled or disabled
* Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable
* Responses to feedback, including making testStdinPipe helper log details of errors copying data to stdin.
Note: We cannot call t.Fatal from a non-test goroutine.
* Use Errorf instead
* Allow backend state files to not include version data when a builtin or reattached provider is in use.
* Add clarifying comment about re-attached providers when finding the matching entry in required_providers
* Report that the default workspace was created to the view
* Refactor: use error comparison via `errors.Is` to identify when no workspaces exist.
* Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseInit func.
* Validate that PSS-related flags can only be used when experiments are enabled, enforce coupling of PSS-related flags when in use.
* Slight rewording of output message about default workspace
* Update test to assert new output about default workspace
* Add ability to parse backend blocks present in a test file's run blocks, validate configuration (#36541)
* Add ability to parse backend blocks from a run block
* Add validation to avoid multiple backend blocks across run blocks that use the same internal state file. Update tests.
* Add validation to avoid multiple backend blocks within a single run block. Update tests.
* Remove use of quotes in diagnostic messages
* Add validation to avoid backend blocks being used in plan run blocks. Update tests.
* Correct local backend blocks in new test fixtures
* Add test to show that different test files can use same backend block for same state key.
* Add validation to enforce state-storage backend types are used
* Remove TODO comment
We only need to consider one file at a time when checking if a state_key already has a backend associated with it; parallelism in `terraform test` is scoped down to individual files.
* Add validation to assert that the backend block must be in the first apply command for an internal state
* Consolidate backend block validation inside a single if statement
* Add initial version of validation that ensures a backend isn't re-used within a file
* Explicitly set the state_key at the point of parsing the config
TODO: What should be done with method (moduletest.Run).GetStateKey?
* Update test fixture now that reusing backend configs has been made invalid
* Add automated test showing validation of reused configuration blocks
* Skip test due to flakiness, minor change to test config naming
* Update test so it tolerates non-deterministic order run blocks are evaluated in
* Remove unnecessary value assignment to r.StateKey
* Replace use of GetStateKey() with accessing the state key that's now set during test config parsing
* Fix bug so that run blocks using child modules get the correct state key set at parsing time
* Update acceptance test to also cover scenario where root and child module state keys are in use
* Update test name
* Add newline to regex
* Ensure consistent place where repeat backend error is raised from
* Write leftover test state(s) to file (#36614)
* Add additional validation that the backend used in a run is a supported type (#36648)
* Prevent test run when leftover state data is present (#36685)
* `test`: Set the initial state for a state files from a backend, allow the run that defines a backend to write state to the backend (#36646)
* Allow use of backend block to set initial state for a state key
* Note about alternative place to keep 'backend factories'
* Allow the run block defining the backend to write state to it
* Fix rebase
* Change to accessing backend init functions via ContextOpts
* Add tests demonstrating how runs containing backend blocks use and update persisted state
* Fix test fixture
* Address test failure due to trouble opening the state file
This problem doesn't happen on MacOS, so I assume is due to the Linux environment of GitHub runners.
* Fix issue with paths properly
I hope
* Fix defect in test assertion
* Pivot back to approach introduced in 4afc3d7
* Let failing tests write to persistent state, add test case covering that.
I split the acceptance tests into happy/unhappy paths for this, which required some of the helper functions' declarations to be raised up to package-level.
* Change how we update internal state files, so that information about the associated backend is never lost
* Fix UpdateStateFile
* Ensure that the states map set by TestStateTransformer associates a backend with the correct run.
* Misc spelling fixes in comments and a log
* Replace state get/set functions with existing helpers (#36747)
* Replace state get/set functions with existing helpers
* Compare to string representation of state
* Compare to string representation of state
* Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729)
* Add validation to enforce skip_cleanup=false cannot be used with backend blocks (#36857)
* Integrate use of backend blocks in tests with skip_cleanup feature (#36848)
* Fix nil pointer error, update test to not be table-driven
* Make using a backend block implicitly set skip_cleanup to true
* Stop state artefacts being created when a backend is in use and no cleanup errors have occurred
* Return diagnostics so calling code knows if cleanup experienced issues or not
* Update tests to show that when cleanup fails a state artefact is created
* Add comment about why diag not returned
* Bug fix - actually pull in the state from the state manager!
* Split and simplify (?) tests to show the backend block can create and/or reuse prior state
* Update test to use new fixtures, assert about state artefact. Fix nil pointer
* Update test fixture in use, add guardrail for flakiness of forced error during cleanup
* Refactor so resource ID set in only one place
* Add documentation for using a `backend` block during `test` (#36832)
* Add backend as a documented block in a run block
* Add documentation about backend blocks in run blocks.
* Make the relationship between backends and state keys more clear, other improvements
* More test documentation (#36838)
* Terraform Test: cleanup command (#36847)
* Allow cleanup of states that depend on prior runs outputs (#36902)
* terraform test: refactor graph edge calculation
* create fake run block nodes during cleanup operation
* tidy up TODOs
* fix tests
* remove old changes
* Update internal/moduletest/graph/node_state_cleanup.go
Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>
* Improve diagnostics around skip_cleanup conflicts (#37385)
* Improve diagnostics around skip_cleanup conflicts
* remove unused dynamic node
* terraform test: refactor manifest file for simplicity (#37412)
* test: refactor apply and plan functions so no run block is needed
* terraform test: write and load state manifest files
* Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729)
* terraform test: add support for skip_cleanup attr
* terraform test: add cleanup command
* terraform test: add backend blocks
* pause
* fix tests
* remove commented code
* terraform test: make controlling destroy functionality experimental (#37419)
* address comments
* Update internal/moduletest/graph/node_state_cleanup.go
Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>
---------
Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>
* add experimental changelog entries
---------
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>
Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>
* Add the ability to make a hash of state store config
* Add test demonstrating that the provider block doesn't impact the hash of a state_store block
* Make sure test asserts what would happen if the schema DID include the provider block
* Update the Hash method to return diagnostics, ignore nested provider blocks, and validate incoming schema and config
* Update tests to use more representative config, fix code under test as a result
* Update Hash method to return hashes for both the state_store and provider blocks
* Add test cases to cover how required fields are tolerated when making the hash
This is because ENVs may supply those values.
* Fix inaccurate comments
* Add test to show that hashes are consistent and exclude the provider block
* Update backend state file to contain hash of provider block's config
* Fix test to expect a hash for the provider config block.
* Fix bug in DeepCopy method, update test to have better error messages when diffs are detected
* Update test to explicitly check hash values
* Try make test intention clearer
* Improve user feedback when state store schema contains the protected word "provider"
* Update tests
* Update test to test the Hash method in a more true-to-life way
Copy of 04a1201878cd1f6f117c43c43c1ee9d0fc17cec1 by Radek Simko
* Update test to use new approach
* Fix `TestInit_stateStoreBlockIsExperimental` test failure
* Implement `ForPlan` method on `StateStoreConfigState`, add `Planner` interface
* Rename `ForPlan` method to `Plan`
* Allow plan files to contain information about state stores
* Add code needed for representing a state store in the Plan struct, which is used when handling plan files
* Add ability to read/write either a backend or state store's data in a plan file. Add some test coverage.
* Update plan's `ProviderAddrs` method to include the provider used for PSS, if present
* Split interfaces
* Apply feedback from code review
* Refactor `SetVersion` to use appropriate constructor
* Split `ProviderAddrs` method test into two
* Fix method call after rename
* Fix test
* Remove change to `(p *Plan) ProviderAddrs()`
We may re-add this when we implement PSS for use during apply commands with plan files
* Remove changes to test, now that the plan doesn't report the provider used for PSS anymore.
* Split code for backend state file vs backend state
* Rename BackendState to BackendConfigState
* Spelling error
* Add `StateStorageConfigState` struct as new implementation of new `ConfigState[T any]` interface.
* Split tests for backend state file vs backend config state structs
* Rename StateStorageConfigState to StateStoreConfigState
* Clarify test name, add comments
* Add tests for StateStoreConfigState methods
* Add test showing state_store in JSON is parsed correctly
* Add detection of malformed backend state files that contain both backend and state_store fields
* Add validation that stops a backend state file being written if it will contain state for both backend and state_store blocks
* Rename `state_storage` to `state_store`
* Rename `state_storage` to `state_store` in filenames
* Move`ConfigState` to its own file
* Fix test name, remove whitespace
* Update `StateStoreConfigState` comment using review suggestion
* Update error message to no longer allude to the environment TF is being run in
* Update the state_store state to use `version.Version` and an adapted version of `tfaddr.Provider` for marshalling version and source data
* Update test helper so it doesn't accidentally supply validation in tests
* Add protection against saving an empty backend state file
* Remove direct testing of (s *Source) MarshalText() and UnmarshalText() methods
* Add Validate method to StateStoreConfigState, use in backend state encoding logic
* Refactor to use new features in registry dependency
Historically (before there were "backends") Terraform had a single state
file format used both for real state snapshots and for tracking where
remote state was stored.
Terraform v0.12 caused these two to be split because we adopted state
snapshot version 4 for real snapshots but retained a subset of version 3
for tracking the remote backend configuration in the local working
directory.
Unfortunately we previously kept that working by retaining a snapshot of
the entire Terraform v0.11 "terraform" package as legacy/terraform, which
happened to still be around because we also needed to retain a copy of
the entire legacy SDK to keep the remote state backends working.
This now hoists just the tiny slice of legacy terraform package
functionality needed to implement the "backend state" file format into
package workdir. This package is a good home for it because it's part of
the working directory state. Ideally it would be accessed through methods
of the workdir.Dir type, but that's too disruptive a refactor to combine
into this and so that'll need to wait for another day; for now we'll keep
the existing callers doing their access through our "clistate" package
that is itself a forked snapshot of what statemgr.Filesystem used to be
in Terraform v0.11.
This removes all but one of the uses of "legacy/terraform" aside from calls
in the other packages under "legacy". We'll clean up the last one in a
later commit, because it's not related to the backend state file format.
Go 1.19's "fmt" has some awareness of the new doc comment formatting
conventions and adjusts the presentation of the source comments to make
it clearer how godoc would interpret them. Therefore this commit includes
various updates made by "go fmt" to acheve that.
In line with our usual convention that we make stylistic/grammar/spelling
tweaks typically only when we're "in the area" changing something else
anyway, I also took this opportunity to review most of the comments that
this updated to see if there were any other opportunities to improve them.
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Thus far our various interactions with the bits of state we keep
associated with a working directory have all been implemented directly
inside the "command" package -- often in the huge command.Meta type -- and
not managed collectively via a single component.
There's too many little codepaths reading and writing from the working
directory and data directory to refactor it all in one step, but this is
an attempt at a first step towards a future where everything that reads
and writes from the current working directory would do so via an object
that encapsulates the implementation details and offers a high-level API
to read and write all of these session-persistent settings.
The design here continues our gradual path towards using a dependency
injection style where "package main" is solely responsible for directly
interacting with the OS command line, the OS environment, the OS working
directory, the stdio streams, and the CLI configuration, and then
communicating the resulting information to the rest of Terraform by wiring
together objects. It seems likely that eventually we'll have enough wiring
code in package main to justify a more explicit organization of that code,
but for this commit the new "workdir.Dir" object is just wired directly in
place of its predecessors, without any significant change of code
organization at that top layer.
This first commit focuses on the main files and directories we use to
find provider plugins, because a subsequent commit will lightly reorganize
the separation of concerns for plugin launching with a similar goal of
collecting all of the relevant logic together into one spot.
