terraform/internal/states/instance_object_src.go

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

218 lines
7.7 KiB
Go
Raw Permalink Normal View History

// Copyright IBM Corp. 2014, 2026
// SPDX-License-Identifier: BUSL-1.1
package states
import (
"bytes"
"fmt"
"reflect"
"github.com/zclconf/go-cty/cty"
ctyjson "github.com/zclconf/go-cty/cty/json"
"github.com/hashicorp/terraform/internal/addrs"
"github.com/hashicorp/terraform/internal/configs/hcl2shim"
Handle marks a little more consistently In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.
2024-04-16 20:20:33 -04:00
"github.com/hashicorp/terraform/internal/lang/marks"
"github.com/hashicorp/terraform/internal/providers"
)
// ResourceInstanceObjectSrc is a not-fully-decoded version of
// ResourceInstanceObject. Decoding of it can be completed by first handling
// any schema migration steps to get to the latest schema version and then
// calling method Decode with the implied type of the latest schema.
type ResourceInstanceObjectSrc struct {
// SchemaVersion is the resource-type-specific schema version number that
// was current when either AttrsJSON or AttrsFlat was encoded. Migration
// steps are required if this is less than the current version number
// reported by the corresponding provider.
SchemaVersion uint64
// AttrsJSON is a JSON-encoded representation of the object attributes,
// encoding the value (of the object type implied by the associated resource
// type schema) that represents this remote object in Terraform Language
// expressions, and is compared with configuration when producing a diff.
//
// This is retained in JSON format here because it may require preprocessing
// before decoding if, for example, the stored attributes are for an older
// schema version which the provider must upgrade before use. If the
// version is current, it is valid to simply decode this using the
// type implied by the current schema, without the need for the provider
// to perform an upgrade first.
//
// When writing a ResourceInstanceObject into the state, AttrsJSON should
// always be conformant to the current schema version and the current
// schema version should be recorded in the SchemaVersion field.
AttrsJSON []byte
IdentitySchemaVersion uint64
IdentityJSON []byte
// AttrsFlat is a legacy form of attributes used in older state file
// formats, and in the new state format for objects that haven't yet been
// upgraded. This attribute is mutually exclusive with Attrs: for any
// ResourceInstanceObject, only one of these attributes may be populated
// and the other must be nil.
//
// An instance object with this field populated should be upgraded to use
// Attrs at the earliest opportunity, since this legacy flatmap-based
// format will be phased out over time. AttrsFlat should not be used when
// writing new or updated objects to state; instead, callers must follow
// the recommendations in the AttrsJSON documentation above.
AttrsFlat map[string]string
Store sensitive attribute paths in state (#26338) * Add creation test and simplify in-place test * Add deletion test * Start adding marking from state Start storing paths that should be marked when pulled out of state. Implements deep copy for attr paths. This commit also includes some comment noise from investigations, and fixing the diff test * Fix apply stripping marks * Expand diff tests * Basic apply test * Update comments on equality checks to clarify current understanding * Add JSON serialization for sensitive paths We need to serialize a slice of cty.Path values to be used to re-mark the sensitive values of a resource instance when loading the state file. Paths consist of a list of steps, each of which may be either getting an attribute value by name, or indexing into a collection by string or number. To serialize these without building a complex parser for a compact string form, we render a nested array of small objects, like so: [ [ { type: "get_attr", value: "foo" }, { type: "index", value: { "type": "number", "value": 2 } } ] ] The above example is equivalent to a path `foo[2]`. * Format diffs with map types Comparisons need unmarked values to operate on, so create unmarked values for those operations. Additionally, change diff to cover map types * Remove debugging printing * Fix bug with marking non-sensitive values When pulling a sensitive value from state, we were previously using those marks to remark the planned new value, but that new value might *not* be sensitive, so let's not do that * Fix apply test Apply was not passing the second state through to the third pass at apply * Consistency in checking for length of paths vs inspecting into value * In apply, don't mark with before paths * AttrPaths test coverage for DeepCopy * Revert format changes Reverts format changes in format/diff for this branch so those changes can be discussed on a separate PR * Refactor name of AttrPaths to AttrSensitivePaths * Rename AttributePaths/attributePaths for naming consistency Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
// AttrSensitivePaths is an array of paths to mark as sensitive coming out of
// state, or to save as sensitive paths when saving state
Handle marks a little more consistently In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.
2024-04-16 20:20:33 -04:00
AttrSensitivePaths []cty.Path
Store sensitive attribute paths in state (#26338) * Add creation test and simplify in-place test * Add deletion test * Start adding marking from state Start storing paths that should be marked when pulled out of state. Implements deep copy for attr paths. This commit also includes some comment noise from investigations, and fixing the diff test * Fix apply stripping marks * Expand diff tests * Basic apply test * Update comments on equality checks to clarify current understanding * Add JSON serialization for sensitive paths We need to serialize a slice of cty.Path values to be used to re-mark the sensitive values of a resource instance when loading the state file. Paths consist of a list of steps, each of which may be either getting an attribute value by name, or indexing into a collection by string or number. To serialize these without building a complex parser for a compact string form, we render a nested array of small objects, like so: [ [ { type: "get_attr", value: "foo" }, { type: "index", value: { "type": "number", "value": 2 } } ] ] The above example is equivalent to a path `foo[2]`. * Format diffs with map types Comparisons need unmarked values to operate on, so create unmarked values for those operations. Additionally, change diff to cover map types * Remove debugging printing * Fix bug with marking non-sensitive values When pulling a sensitive value from state, we were previously using those marks to remark the planned new value, but that new value might *not* be sensitive, so let's not do that * Fix apply test Apply was not passing the second state through to the third pass at apply * Consistency in checking for length of paths vs inspecting into value * In apply, don't mark with before paths * AttrPaths test coverage for DeepCopy * Revert format changes Reverts format changes in format/diff for this branch so those changes can be discussed on a separate PR * Refactor name of AttrPaths to AttrSensitivePaths * Rename AttributePaths/attributePaths for naming consistency Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
// These fields all correspond to the fields of the same name on
// ResourceInstanceObject.
Private []byte
Status ObjectStatus
Dependencies []addrs.ConfigResource
CreateBeforeDestroy bool
// decodeValueCache stored the decoded value for repeated decodings.
decodeValueCache cty.Value
// decodeIdentityCache stored the decoded identity for repeated decodings.
decodeIdentityCache cty.Value
}
// Decode unmarshals the raw representation of the object attributes. Pass the
// schema of the corresponding resource type for correct operation.
//
// Before calling Decode, the caller must check that the SchemaVersion field
// exactly equals the version number of the schema whose implied type is being
// passed, or else the result is undefined.
//
// If the object has an identity, the schema must also contain a resource
// identity schema for the identity to be decoded.
//
// The returned object may share internal references with the receiver and
// so the caller must not mutate the receiver any further once once this
// method is called.
func (os *ResourceInstanceObjectSrc) Decode(schema providers.Schema) (*ResourceInstanceObject, error) {
var val cty.Value
var err error
attrsTy := schema.Body.ImpliedType()
switch {
case os.decodeValueCache != cty.NilVal:
val = os.decodeValueCache
case os.AttrsFlat != nil:
// Legacy mode. We'll do our best to unpick this from the flatmap.
val, err = hcl2shim.HCL2ValueFromFlatmap(os.AttrsFlat, attrsTy)
if err != nil {
return nil, err
}
default:
val, err = ctyjson.Unmarshal(os.AttrsJSON, attrsTy)
Handle marks a little more consistently In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.
2024-04-16 20:20:33 -04:00
val = marks.MarkPaths(val, marks.Sensitive, os.AttrSensitivePaths)
if err != nil {
return nil, err
}
}
var identity cty.Value
if os.decodeIdentityCache != cty.NilVal {
identity = os.decodeIdentityCache
} else if os.IdentityJSON != nil && schema.Identity != nil {
identity, err = ctyjson.Unmarshal(os.IdentityJSON, schema.Identity.ImpliedType())
if err != nil {
return nil, fmt.Errorf("failed to decode identity: %s. This is most likely a bug in the Provider, providers must not change the identity schema without updating the identity schema version", err.Error())
}
}
return &ResourceInstanceObject{
Value: val,
Identity: identity,
Status: os.Status,
Dependencies: os.Dependencies,
Private: os.Private,
CreateBeforeDestroy: os.CreateBeforeDestroy,
}, nil
}
// CompleteUpgrade creates a new ResourceInstanceObjectSrc by copying the
// metadata from the receiver and writing in the given new schema version
// and attribute value that are presumed to have resulted from upgrading
// from an older schema version.
func (os *ResourceInstanceObjectSrc) CompleteUpgrade(newAttrs cty.Value, newType cty.Type, newSchemaVersion uint64) (*ResourceInstanceObjectSrc, error) {
new := os.DeepCopy()
new.AttrsFlat = nil // We always use JSON after an upgrade, even if the source used flatmap
// This is the same principle as ResourceInstanceObject.Encode, but
// avoiding a decode/re-encode cycle because we don't have type info
// available for the "old" attributes.
newAttrs = cty.UnknownAsNull(newAttrs)
src, err := ctyjson.Marshal(newAttrs, newType)
if err != nil {
return nil, err
}
new.AttrsJSON = src
new.SchemaVersion = newSchemaVersion
return new, nil
}
func (os *ResourceInstanceObjectSrc) CompleteIdentityUpgrade(newAttrs cty.Value, schema providers.Schema) (*ResourceInstanceObjectSrc, error) {
new := os.DeepCopy()
src, err := ctyjson.Marshal(newAttrs, schema.Identity.ImpliedType())
if err != nil {
return nil, err
}
new.IdentityJSON = src
new.IdentitySchemaVersion = uint64(schema.IdentityVersion)
return new, nil
}
// Equal compares two ResourceInstanceObjectSrc objects for equality, skipping
// any internal fields which are not stored to the final serialized state.
func (os *ResourceInstanceObjectSrc) Equal(other *ResourceInstanceObjectSrc) bool {
if os == other {
return true
}
if os == nil || other == nil {
return false
}
if os.SchemaVersion != other.SchemaVersion {
return false
}
if os.IdentitySchemaVersion != other.IdentitySchemaVersion {
return false
}
if os.Status != other.Status {
return false
}
if os.CreateBeforeDestroy != other.CreateBeforeDestroy {
return false
}
if !bytes.Equal(os.AttrsJSON, other.AttrsJSON) {
return false
}
if !bytes.Equal(os.IdentityJSON, other.IdentityJSON) {
return false
}
if !bytes.Equal(os.Private, other.Private) {
return false
}
// Compare legacy AttrsFlat maps. We shouldn't see this ever being used, but
// deal with in just in case until we remove it entirely. These are all
// simple maps of strings, so DeepEqual is perfectly fine here.
if !reflect.DeepEqual(os.AttrsFlat, other.AttrsFlat) {
return false
}
// We skip fields that have no functional impact on resource state.
return true
}