In the very first implementation of "sensitive values" we were
unfortunately not disciplined about separating the idea of "marked value"
from the idea of "sensitive value" (where the latter is a subset of the
former). The first implementation just assumed that any marking whatsoever
meant "sensitive".
We later improved that by adding the marks package and the marks.Sensitive
value to standardize on the representation of "sensitive value" as being
a value marked with _that specific mark_.
However, we did not perform a thorough review of all of the mark-handling
codepaths to make sure they all agreed on that definition. In particular,
the state and plan models were both designed as if they supported arbitrary
marks but then in practice marks other than marks.Sensitive would be
handled in various inconsistent ways: dropped entirely, or interpreted as
if marks.Sensitive, and possibly do so inconsistently when a value is
used only in memory vs. round-tripped through a wire/file format.
The goal of this commit is to resolve those oddities so that there are now
two possible situations:
- General mark handling: some codepaths genuinely handle marks
generically, by transporting them from input value to output value in
a way consistent with how cty itself deals with marks. This is the
ideal case because it means we can add new marks in future and assume
these codepaths will handle them correctly without any further
modifications.
- Sensitive-only mark preservation: the codepaths that interact with our
wire protocols and file formats typically have only specialized support
for sensitive values in particular, and lack support for any other
marks. Those codepaths are now subject to a new rule where they must
return an error if asked to deal with any other mark, so that if we
introduce new marks in future we'll be forced either to define how we'll
avoid those markings reaching the file/wire formats or extend the
file/wire formats to support the new marks.
Some new helper functions in package marks are intended to standardize how
we deal with the "sensitive values only" situations, in the hope that
this will make it easier to keep things consistent as the codebase evolves
in future.
In practice the modules runtime only ever uses marks.Sensitive as a mark
today, so all of these checks are effectively covering "should never
happen" cases. The only other mark Terraform uses is an implementation
detail of "terraform console" and does not interact with any of the
codepaths that only support sensitive values in particular.
Previously we were directly using the planproto.DynamicValue message type,
but that's symmetrical with plans.DynamicValue in that it encodes only
the value itself and not associated metadata such as the paths that have
sensitive value marks.
This new tfstackdata1.DynamicValue therefore echoes
terraform1.DynamicValue by carrying both the value and the metadata
together.
This implies a slight change to one of the existing message types that
was encoding a map of planned input values for each component instance,
but in practice we're not currently making use of that data anyway -- it's
there to enable a hypothetical extra correctness check in the stacks
runtime so that we can detect inconsistency problems closer to their
source -- and so this change doesn't affect anything in practice. Before
we make use of it we'll probably want to change the internal API a little
so that we can preserve the sensitive value metadata on those values, but
that's beyond the scope of this PR.
The main point of adding this is to track the output values for a component
instance as part of the raw state, and so the new field for that is also
added here although nothing will read or write it yet. Actual use of that
new field will follow in future commits.
We need to retain the prior state (in a form that Terraform Core's modules
runtime can accept) between the plan and apply phases because Terraform
uses it to verify that the provider's "final plan" is consistent with
the initial plan.
We previously tried to do this by reusing the "old" value from the planned
change, but that's not really possible in practice because of the
historical implementation detail that Terraform wants state information
in a JSON format and plan information in MessagePack.
This also contains the beginnings of handling the "discarding" of
unrecognized keys from the state data structures, although we'll probably
need to do more in that area later since this is just the bare minimum.
Previously we just had a stub that only generated basic external
description objects. Now we'll also emit "raw state" objects for these,
and have some initial support for deposed objects too.
