terraform/internal/command/workdir/dir.go

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

159 lines
6.8 KiB
Go
Raw Permalink Normal View History

// Copyright IBM Corp. 2014, 2026
// SPDX-License-Identifier: BUSL-1.1
workdir: Start of a new package for working directory state management Thus far our various interactions with the bits of state we keep associated with a working directory have all been implemented directly inside the "command" package -- often in the huge command.Meta type -- and not managed collectively via a single component. There's too many little codepaths reading and writing from the working directory and data directory to refactor it all in one step, but this is an attempt at a first step towards a future where everything that reads and writes from the current working directory would do so via an object that encapsulates the implementation details and offers a high-level API to read and write all of these session-persistent settings. The design here continues our gradual path towards using a dependency injection style where "package main" is solely responsible for directly interacting with the OS command line, the OS environment, the OS working directory, the stdio streams, and the CLI configuration, and then communicating the resulting information to the rest of Terraform by wiring together objects. It seems likely that eventually we'll have enough wiring code in package main to justify a more explicit organization of that code, but for this commit the new "workdir.Dir" object is just wired directly in place of its predecessors, without any significant change of code organization at that top layer. This first commit focuses on the main files and directories we use to find provider plugins, because a subsequent commit will lightly reorganize the separation of concerns for plugin launching with a similar goal of collecting all of the relevant logic together into one spot.
2021-09-01 20:01:44 -04:00
package workdir
import (
"fmt"
"os"
"path/filepath"
)
// Dir represents a single Terraform working directory.
//
// "Working directory" is unfortunately a slight misnomer, because non-default
// options can potentially stretch the definition such that multiple working
// directories end up appearing to share a data directory, or other similar
Implement controlling destroy functionality within Terraform Test (#37359) * Add ability to parse backend blocks present in a test file's run blocks, validate configuration (#36541) * Add ability to parse backend blocks from a run block * Add validation to avoid multiple backend blocks across run blocks that use the same internal state file. Update tests. * Add validation to avoid multiple backend blocks within a single run block. Update tests. * Remove use of quotes in diagnostic messages * Add validation to avoid backend blocks being used in plan run blocks. Update tests. * Correct local backend blocks in new test fixtures * Add test to show that different test files can use same backend block for same state key. * Add validation to enforce state-storage backend types are used * Remove TODO comment We only need to consider one file at a time when checking if a state_key already has a backend associated with it; parallelism in `terraform test` is scoped down to individual files. * Add validation to assert that the backend block must be in the first apply command for an internal state * Consolidate backend block validation inside a single if statement * Add initial version of validation that ensures a backend isn't re-used within a file * Explicitly set the state_key at the point of parsing the config TODO: What should be done with method (moduletest.Run).GetStateKey? * Update test fixture now that reusing backend configs has been made invalid * Add automated test showing validation of reused configuration blocks * Skip test due to flakiness, minor change to test config naming * Update test so it tolerates non-deterministic order run blocks are evaluated in * Remove unnecessary value assignment to r.StateKey * Replace use of GetStateKey() with accessing the state key that's now set during test config parsing * Fix bug so that run blocks using child modules get the correct state key set at parsing time * Update acceptance test to also cover scenario where root and child module state keys are in use * Update test name * Add newline to regex * Ensure consistent place where repeat backend error is raised from * Write leftover test state(s) to file (#36614) * Add additional validation that the backend used in a run is a supported type (#36648) * Prevent test run when leftover state data is present (#36685) * `test`: Set the initial state for a state files from a backend, allow the run that defines a backend to write state to the backend (#36646) * Allow use of backend block to set initial state for a state key * Note about alternative place to keep 'backend factories' * Allow the run block defining the backend to write state to it * Fix rebase * Change to accessing backend init functions via ContextOpts * Add tests demonstrating how runs containing backend blocks use and update persisted state * Fix test fixture * Address test failure due to trouble opening the state file This problem doesn't happen on MacOS, so I assume is due to the Linux environment of GitHub runners. * Fix issue with paths properly I hope * Fix defect in test assertion * Pivot back to approach introduced in 4afc3d7 * Let failing tests write to persistent state, add test case covering that. I split the acceptance tests into happy/unhappy paths for this, which required some of the helper functions' declarations to be raised up to package-level. * Change how we update internal state files, so that information about the associated backend is never lost * Fix UpdateStateFile * Ensure that the states map set by TestStateTransformer associates a backend with the correct run. * Misc spelling fixes in comments and a log * Replace state get/set functions with existing helpers (#36747) * Replace state get/set functions with existing helpers * Compare to string representation of state * Compare to string representation of state * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * Add validation to enforce skip_cleanup=false cannot be used with backend blocks (#36857) * Integrate use of backend blocks in tests with skip_cleanup feature (#36848) * Fix nil pointer error, update test to not be table-driven * Make using a backend block implicitly set skip_cleanup to true * Stop state artefacts being created when a backend is in use and no cleanup errors have occurred * Return diagnostics so calling code knows if cleanup experienced issues or not * Update tests to show that when cleanup fails a state artefact is created * Add comment about why diag not returned * Bug fix - actually pull in the state from the state manager! * Split and simplify (?) tests to show the backend block can create and/or reuse prior state * Update test to use new fixtures, assert about state artefact. Fix nil pointer * Update test fixture in use, add guardrail for flakiness of forced error during cleanup * Refactor so resource ID set in only one place * Add documentation for using a `backend` block during `test` (#36832) * Add backend as a documented block in a run block * Add documentation about backend blocks in run blocks. * Make the relationship between backends and state keys more clear, other improvements * More test documentation (#36838) * Terraform Test: cleanup command (#36847) * Allow cleanup of states that depend on prior runs outputs (#36902) * terraform test: refactor graph edge calculation * create fake run block nodes during cleanup operation * tidy up TODOs * fix tests * remove old changes * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * Improve diagnostics around skip_cleanup conflicts (#37385) * Improve diagnostics around skip_cleanup conflicts * remove unused dynamic node * terraform test: refactor manifest file for simplicity (#37412) * test: refactor apply and plan functions so no run block is needed * terraform test: write and load state manifest files * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * terraform test: add support for skip_cleanup attr * terraform test: add cleanup command * terraform test: add backend blocks * pause * fix tests * remove commented code * terraform test: make controlling destroy functionality experimental (#37419) * address comments * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * add experimental changelog entries --------- Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com> Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>
2025-09-10 11:22:20 -04:00
// anomalies, but we continue to use this terminology both for historical
workdir: Start of a new package for working directory state management Thus far our various interactions with the bits of state we keep associated with a working directory have all been implemented directly inside the "command" package -- often in the huge command.Meta type -- and not managed collectively via a single component. There's too many little codepaths reading and writing from the working directory and data directory to refactor it all in one step, but this is an attempt at a first step towards a future where everything that reads and writes from the current working directory would do so via an object that encapsulates the implementation details and offers a high-level API to read and write all of these session-persistent settings. The design here continues our gradual path towards using a dependency injection style where "package main" is solely responsible for directly interacting with the OS command line, the OS environment, the OS working directory, the stdio streams, and the CLI configuration, and then communicating the resulting information to the rest of Terraform by wiring together objects. It seems likely that eventually we'll have enough wiring code in package main to justify a more explicit organization of that code, but for this commit the new "workdir.Dir" object is just wired directly in place of its predecessors, without any significant change of code organization at that top layer. This first commit focuses on the main files and directories we use to find provider plugins, because a subsequent commit will lightly reorganize the separation of concerns for plugin launching with a similar goal of collecting all of the relevant logic together into one spot.
2021-09-01 20:01:44 -04:00
// reasons and because it reflects the common case without any special
// overrides.
//
// The naming convention for methods on this type is that methods whose names
// begin with "Override" affect only characteristics of the particular object
// they're called on, changing where it looks for data, while methods whose
// names begin with "Set" will write settings to disk such that other instances
// referring to the same directories will also see them. Given that, the
// "Override" methods should be used only during the initialization steps
// for a Dir object, typically only inside "package main", so that all
// subsequent work elsewhere will access consistent locations on disk.
//
// We're gradually transitioning to using this type to manage working directory
// settings, and so not everything in the working directory "data dir" is
// encapsulated here yet, but hopefully we'll gradually migrate all of those
// settings here over time. The working directory state not yet managed in here
// is typically managed directly in the "command" package, either directly
// inside commands or in methods of the giant command.Meta type.
type Dir struct {
// mainDir is the path to the directory that we present as the
// "working directory" in the user model, which is typically the
// current working directory when running Terraform CLI, or the
// directory explicitly chosen by the user using the -chdir=...
// global option.
mainDir string
// originalDir is the path to the working directory that was
// selected when creating the Terraform CLI process, regardless of
// -chdir=... being set. This is only for very limited purposes
// related to backward compatibility; most functionality should
// use mainDir instead.
originalDir string
// dataDir is the path to the directory where we will store our
// working directory settings and artifacts. This is typically a
// directory named ".terraform" within mainDir, but users may
// override it.
dataDir string
}
// NewDir constructs a new working directory, anchored at the given path.
//
// In normal use, mainPath should be "." to reflect the current working
// directory, with "package main" having switched the process's current
// working directory if necessary prior to calling this function. However,
// unusual situations in tests may set mainPath to a temporary directory, or
// similar.
//
// WARNING: Although the logic in this package is intended to work regardless
// of whether mainPath is actually the current working directory, we're
// currently in a transitional state where this package shares responsibility
// for the working directory with various command.Meta methods, and those
// often assume that the main path of the working directory will always be
// ".". If you're writing test code that spans across both areas of
// responsibility then you must ensure that the test temporarily changes the
// test process's working directory to the directory returned by RootModuleDir
// before using the result inside a command.Meta.
func NewDir(mainPath string) *Dir {
mainPath = filepath.Clean(mainPath)
return &Dir{
mainDir: mainPath,
originalDir: mainPath,
dataDir: filepath.Join(mainPath, ".terraform"),
}
}
// OverrideOriginalWorkingDir records a different path as the
// "original working directory" for the reciever.
//
// Use this only to record the original working directory when Terraform is run
// with the -chdir=... global option. In that case, the directory given in
// -chdir=... is the "main path" to pass in to NewDir, while the original
// working directory should be sent to this method.
func (d *Dir) OverrideOriginalWorkingDir(originalPath string) {
d.originalDir = filepath.Clean(originalPath)
}
// OverrideDataDir chooses a specific alternative directory to read and write
// the persistent working directory settings.
//
// "package main" can call this if it detects that the user has overridden
// the default location by setting the relevant environment variable. Don't
// call this when that environment variable isn't set, in order to preserve
// the default setting of a dot-prefixed directory directly inside the main
// working directory.
func (d *Dir) OverrideDataDir(dataDir string) {
d.dataDir = filepath.Clean(dataDir)
}
// RootModuleDir returns the directory where we expect to find the root module
// configuration for this working directory.
func (d *Dir) RootModuleDir() string {
// The root module configuration is just directly inside the main directory.
return d.mainDir
}
// OriginalWorkingDir returns the true, operating-system-originated working
// directory that the current Terraform process was launched from.
//
// This is usually the same as the main working directory, but differs in the
// special case where the user ran Terraform with the global -chdir=...
// option. This is here only for a few backward compatibility affordances
// from before we had the -chdir=... option, so should typically not be used
// for anything new.
func (d *Dir) OriginalWorkingDir() string {
return d.originalDir
}
// DataDir returns the base path where the reciever keeps all of the settings
// and artifacts that must persist between consecutive commands in a single
// session.
//
// This is exported only to allow the legacy behaviors in command.Meta to
// continue accessing this directory directly. Over time we should replace
// all of those direct accesses with methods on this type, and then remove
// this method. Avoid using this method for new use-cases.
func (d *Dir) DataDir() string {
return d.dataDir
}
Implement controlling destroy functionality within Terraform Test (#37359) * Add ability to parse backend blocks present in a test file's run blocks, validate configuration (#36541) * Add ability to parse backend blocks from a run block * Add validation to avoid multiple backend blocks across run blocks that use the same internal state file. Update tests. * Add validation to avoid multiple backend blocks within a single run block. Update tests. * Remove use of quotes in diagnostic messages * Add validation to avoid backend blocks being used in plan run blocks. Update tests. * Correct local backend blocks in new test fixtures * Add test to show that different test files can use same backend block for same state key. * Add validation to enforce state-storage backend types are used * Remove TODO comment We only need to consider one file at a time when checking if a state_key already has a backend associated with it; parallelism in `terraform test` is scoped down to individual files. * Add validation to assert that the backend block must be in the first apply command for an internal state * Consolidate backend block validation inside a single if statement * Add initial version of validation that ensures a backend isn't re-used within a file * Explicitly set the state_key at the point of parsing the config TODO: What should be done with method (moduletest.Run).GetStateKey? * Update test fixture now that reusing backend configs has been made invalid * Add automated test showing validation of reused configuration blocks * Skip test due to flakiness, minor change to test config naming * Update test so it tolerates non-deterministic order run blocks are evaluated in * Remove unnecessary value assignment to r.StateKey * Replace use of GetStateKey() with accessing the state key that's now set during test config parsing * Fix bug so that run blocks using child modules get the correct state key set at parsing time * Update acceptance test to also cover scenario where root and child module state keys are in use * Update test name * Add newline to regex * Ensure consistent place where repeat backend error is raised from * Write leftover test state(s) to file (#36614) * Add additional validation that the backend used in a run is a supported type (#36648) * Prevent test run when leftover state data is present (#36685) * `test`: Set the initial state for a state files from a backend, allow the run that defines a backend to write state to the backend (#36646) * Allow use of backend block to set initial state for a state key * Note about alternative place to keep 'backend factories' * Allow the run block defining the backend to write state to it * Fix rebase * Change to accessing backend init functions via ContextOpts * Add tests demonstrating how runs containing backend blocks use and update persisted state * Fix test fixture * Address test failure due to trouble opening the state file This problem doesn't happen on MacOS, so I assume is due to the Linux environment of GitHub runners. * Fix issue with paths properly I hope * Fix defect in test assertion * Pivot back to approach introduced in 4afc3d7 * Let failing tests write to persistent state, add test case covering that. I split the acceptance tests into happy/unhappy paths for this, which required some of the helper functions' declarations to be raised up to package-level. * Change how we update internal state files, so that information about the associated backend is never lost * Fix UpdateStateFile * Ensure that the states map set by TestStateTransformer associates a backend with the correct run. * Misc spelling fixes in comments and a log * Replace state get/set functions with existing helpers (#36747) * Replace state get/set functions with existing helpers * Compare to string representation of state * Compare to string representation of state * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * Add validation to enforce skip_cleanup=false cannot be used with backend blocks (#36857) * Integrate use of backend blocks in tests with skip_cleanup feature (#36848) * Fix nil pointer error, update test to not be table-driven * Make using a backend block implicitly set skip_cleanup to true * Stop state artefacts being created when a backend is in use and no cleanup errors have occurred * Return diagnostics so calling code knows if cleanup experienced issues or not * Update tests to show that when cleanup fails a state artefact is created * Add comment about why diag not returned * Bug fix - actually pull in the state from the state manager! * Split and simplify (?) tests to show the backend block can create and/or reuse prior state * Update test to use new fixtures, assert about state artefact. Fix nil pointer * Update test fixture in use, add guardrail for flakiness of forced error during cleanup * Refactor so resource ID set in only one place * Add documentation for using a `backend` block during `test` (#36832) * Add backend as a documented block in a run block * Add documentation about backend blocks in run blocks. * Make the relationship between backends and state keys more clear, other improvements * More test documentation (#36838) * Terraform Test: cleanup command (#36847) * Allow cleanup of states that depend on prior runs outputs (#36902) * terraform test: refactor graph edge calculation * create fake run block nodes during cleanup operation * tidy up TODOs * fix tests * remove old changes * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * Improve diagnostics around skip_cleanup conflicts (#37385) * Improve diagnostics around skip_cleanup conflicts * remove unused dynamic node * terraform test: refactor manifest file for simplicity (#37412) * test: refactor apply and plan functions so no run block is needed * terraform test: write and load state manifest files * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * terraform test: add support for skip_cleanup attr * terraform test: add cleanup command * terraform test: add backend blocks * pause * fix tests * remove commented code * terraform test: make controlling destroy functionality experimental (#37419) * address comments * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * add experimental changelog entries --------- Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com> Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>
2025-09-10 11:22:20 -04:00
// TestDataDir returns the path where the receiver keeps settings
// and artifacts related to terraform tests.
func (d *Dir) TestDataDir() string {
return filepath.Join(d.dataDir, "test")
}
workdir: Start of a new package for working directory state management Thus far our various interactions with the bits of state we keep associated with a working directory have all been implemented directly inside the "command" package -- often in the huge command.Meta type -- and not managed collectively via a single component. There's too many little codepaths reading and writing from the working directory and data directory to refactor it all in one step, but this is an attempt at a first step towards a future where everything that reads and writes from the current working directory would do so via an object that encapsulates the implementation details and offers a high-level API to read and write all of these session-persistent settings. The design here continues our gradual path towards using a dependency injection style where "package main" is solely responsible for directly interacting with the OS command line, the OS environment, the OS working directory, the stdio streams, and the CLI configuration, and then communicating the resulting information to the rest of Terraform by wiring together objects. It seems likely that eventually we'll have enough wiring code in package main to justify a more explicit organization of that code, but for this commit the new "workdir.Dir" object is just wired directly in place of its predecessors, without any significant change of code organization at that top layer. This first commit focuses on the main files and directories we use to find provider plugins, because a subsequent commit will lightly reorganize the separation of concerns for plugin launching with a similar goal of collecting all of the relevant logic together into one spot.
2021-09-01 20:01:44 -04:00
// ensureDataDir creates the data directory and all of the necessary parent
// directories that lead to it, if they don't already exist.
//
// For directories that already exist ensureDataDir will preserve their
// permissions, while it'll create any new directories to be owned by the user
// running Terraform, readable and writable by that user, and readable by
// all other users, or some approximation of that on non-Unix platforms which
// have a different permissions model.
func (d *Dir) ensureDataDir() error {
err := os.MkdirAll(d.dataDir, 0755)
if err != nil {
return fmt.Errorf("failed to prepare working directory: %w", err)
}
return nil
}