The convert function allows for inline type conversions within the
Terraform language. A custom decoder for the function which allows for
type expression literals as the second parameter.
The templatefile function actually calls the full renderer within the
TypeFunc to determine what the resulting might be, but that also
requires unmarking the top-level of the vars map to strip marks even
though the value is unused.
While the evaluator can deal with unknown template variables, if the
entire map of variables is unknown, we can't create the map and need to
short-circuit the call.
Terraform attempts to track marks as accurately as possible, but unknown
values may not always have the same marks as they will when they become
known. This is most easily seen with functions, which are allowed to
return an unknown value when faced with any unknown arguments, while
they are also allowed to manipulate the marks on the values as they see
fit. This results in situations where the marks simply cannot be known.
Terraform generally takes the stance that if an unknown has a mark, it
will remain in the final value, but the absence of a mark is not
indicative of the absence of any marks in the final value.
You can't deal with marks in a function without also dealing with
unknown. If AllowMarked is true, but AllowKnown is not, the caller will
short-circuit the evaluation with an unknown, without being able to deal
with marks from all arguments.
Before a timestamp is assigned during plan, the `time.Time` value will
be the zero value for that type. This represents a case where the
timestamp is not yet known, but the handling of that timestamp isn't
using our usual cty value system for representing unknowns. Rather than
refactor the use of the timestamp itself throughout the code, we can
simply have the function return an unknown if it encounters a zero
value.
This is another part of the existing ephemeral_values experiment, taking
a value of any type that might have ephemeral values in it and returning
a value of the same type which has any ephemeral value replaced with a
null value.
The primary purpose of this is to allow a module to conveniently return an
object that would normally contain nested ephemeral values -- such as an
instance of a managed resource type that has a write-only attribute --
through an output value that isn't declared as ephemeral. This would then
expose all of the non-ephemeral parts of the object but withhold the
ephemeral parts. In the case of write-only attributes, it exposes the
normal attributes while withholding the write-only ones.
The name of this function could potentially change before stabilization,
because it's quite long and clunky. I did originally consider
"nonephemeral" to match with the existing "nonsensitive", but that didn't
feel right because "nonsensitive" removes the sensitive mark while
preserving the underlying value while this function removes the mark and
the real value at the same time. (It would not be appropriate to have a
function that just removes the ephemeral mark while preserving the value,
because correct handling of ephemerality is important for correctness
while sensitivity is primarily a UI concern so we don't need to be quite
so picky about it.)
The templatestring function has some special constraints on its first
argument that are included to add some intentional friction for those who
are new to Terraform, want to do some simple template rendering, but have
only found the templatestring function so far.
We know from previous experience with the hashicorp/template provider that
this sort of functionality tends to attract those who haven't yet learned
that the Terraform language has built-in support for string templates
(without calling any function), who would then get confused by the need
for an extra level of escaping to render a template string only indirectly
through this function.
However, this rule is not intended to be onerous and require writing the
rest of the containing module in an unnatural way to work around it, so
here we loosen the rule to allow some additional forms:
- An index expression whose collection operand meets these rules.
- A relative traversal whose source operand meets these rules.
In particular this makes it possible to write an expression like:
data.example.example[each.key].result
...which is a relative traversal from an index from a scope traversal,
and is a very reasonable thing to write if you've retrieved multiple
templates using a data resource that uses for_each.
This also treats splat expressions in the same way as index expressions
at the static check stage, but that's only to allow us to reach the
dynamic type check that will ultimately report that a string is required,
because the result of a splat expression is a tuple. The type-related
error message is (subjectively) more helpful/relevant than the
syntax-related one for this case.
Finally, this includes some revisions to the documentation for this
function to correct some editing errors from the first pass and to slightly
loosen the language about what's allowed. It's still a little vague about
what exactly is allowed, but I'm doubtful that a precise definition in
terms of HCL's expression types would be very enlightening for a typical
reader anyway. We can tweak the specificity of the language here if we
start to see repeated questions about what is and is not valid.
Previously we were partially propagating any marks from the path, but not
going all the way so we still ran into trouble when trying to use the
string containing the file contents.
Now we'll have loadTmpl also return the marks it had to read through to
actually parse the template, and then we'll use those (instead of the
original path marks) to mark the result. In practice the pathMarks and
the tmplMarks should always match today, but this is intentionally
structured to make the data flow clearer -- the marks always travel along
with whatever they related to -- so we're less likely to break this
accidentally under future maintenence.
This function complements the existing "templatefile" to deal with the
unusual situation of rendering a template that comes from somewhere
outside of the current module's source code, such as from a data resource
result.
We have some historical experience with the now-deprecated
hashicorp/template provider and its template_file data source, where we
found that new authors would find it via web search and assume it was
"the way" to render templates in Terraform, and then get frustrated
dealing with the confusing situation of writing a string template that
generates another string template for a second round of template rendering.
To try to support those who have this unusual need without creating another
attractive nuisance that would derail new authors, this function imposes
the artificial extra rule that its template argument may only be populated
using a single reference to a symbol defined elsewhere in the same module.
This is intended to entice folks trying to use this function for something
other than its intended purpose to refer to its documentation (once
written) and then hopefully learn what other Terraform language feature
they ought to have used instead.
The syntax restriction only goes one level deep, so particularly-determined
authors can still intentionally misuse this function by adding one level
of indirection, such as by building template source code in a local value
and then passing that local value as the template argument. The restriction
is in place only to reduce the chances of someone _misunderstanding_ the
purpose of this function; we don't intend to prevent someone from actively
deciding to misuse it, if they have a good reason to do so.
This new function inherits the same restriction as templatefile where it
does not allow recursively calling other template-rendering functions.
This is to dissuade from trying to use Terraform templates "at large",
since Terraform's template language is not designed for such uses. It would
be better to build a Terraform provider that wraps a more featureful
template system like Gonja if someone really does need advanced templating,
beyond Terraform's basic goals of being able to build small configuration
files, etc.
Because this function's intended purpose is rendering templates obtained
from elsewhere, this function also blocks calls to any of Terraform's
functions that would read from the filesystem of the computer where
Terraform is running. This is a small additional measure of isolation to
reduce the risk of an attacker somehow modifying a dynamically-fetched
template to inspire Terraform to write sensitive data from the host
computer into a location accessible to the same attacker, or similar.
This is currently only a language experiment and so will not yet be
available in stable releases of Terraform. Before stabilizing this and
committing to supporting it indefinitely we'll want to gather feedback on
whether this function actually meets the intended narrow set of use-cases
around dynamic template rendering.
CDKTF uses these definitions to generate docs which end up on developer.hashicorp.com
There are content checks that fail for this link as the product (terraform) is required to be explicitly stated now
Now that we dynamically add scoped versions to the global functions we
can't check the static Descriptions list. This should be OK since the
function descriptions are taken directly from Descriptions list, and if
any other scoped functions were added in the future, they may not be
coming from the current global set of functions anyway.
This essentially doubles up the registrations for all of the built-in
functions to have both non-namespaced and core::-namespaced versions of
each function.
This is in preparation for later commits introducing other namespaces,
such as a provider:: namespace which could hold functions that were
contributed by providers that are currently in scope.
If the string to be tested is an unknown value that's been refined with
a prefix and the prefix we're being asked to test is in turn a prefix of
that known prefix then we can return a known answer despite the inputs
not being fully known.
There are also some other similar deductions we can make about other
combinations of inputs.
This extra analysis could be useful in a custom condition check that
requires a string with a particular prefix, since it can allow the
condition to fail even on partially-unknown input, thereby giving earlier
feedback about a problem.
cty's new "refinements" concept allows us to reduce the range of unknown
values from our functions. This initial changeset focuses only on
declaring which functions are guaranteed to return a non-null result,
which is a helpful baseline refinement because it allows "== null" and
"!= null" tests to produce known results even when the given value is
otherwise unknown.
This commit also includes some updates to test results that are now
refined based on cty's own built-in refinement behaviors, just as a
result of us having updated cty in the previous commit.
We inadvertently incorporated the new minor release of cty into the 1.4
branch, and that's introduced some more refined handling of unknown values
that is too much of a change to introduce in a patch release.
Therefore this reverts back to the previous minor release for the v1.4
series, and then we'll separately get the main branch ready to work
correctly with the new cty before Terraform v1.5.
This reverts just the upgrade and the corresponding test changes from
#32775, while retaining the HCL upgrade and the new test case it
introduced for that bug it was trying to fix. That new test is still
passing so it seems that the cty upgrade is not crucial to that fix.
* Add consolidated function description list
* Add function parameter descriptions
* Add descriptions to all functions
* Add sanity test for function descriptions
* Apply suggestions from code review
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
This is a complement to "timestamp" and "timeadd" which allows
establishing the ordering of two different timestamps while taking into
account their timezone offsets, which isn't otherwise possible using the
existing primitives in the Terraform language.
Go 1.19's "fmt" has some awareness of the new doc comment formatting
conventions and adjusts the presentation of the source comments to make
it clearer how godoc would interpret them. Therefore this commit includes
various updates made by "go fmt" to acheve that.
In line with our usual convention that we make stylistic/grammar/spelling
tweaks typically only when we're "in the area" changing something else
anyway, I also took this opportunity to review most of the comments that
this updated to see if there were any other opportunities to improve them.
We had intended these functions to attempt to convert any given value, but
there is a special behavior in the function system where functions must
opt in to being able to handle dynamically-typed arguments so that we
don't need to repeat the special case for that inside every function
implementation.
In this case we _do_ want to specially handle dynamically-typed values,
because the keyword "null" in HCL produces
cty.NullVal(cty.DynamicPseudoType) and we want the conversion function
to convert it to a null of a more specific type.
These conversion functions are already just a thin wrapper around the
underlying type conversion functionality anyway, and that already supports
converting dynamic-typed values in the expected way, so we can just opt
in to allowing dynamically-typed values and let the conversion
functionality do the expected work.
Fixing this allows module authors to use type conversion functions to
give additional type information to Terraform in situations that are too
ambiguous to be handled automatically by the type inference/unification
process. Previously tostring(null) was effectively a no-op, totally
ignoring the author's request to treat the null as a string.
The sum() function accepts a collection of values which must all convert
to numbers. It is valid for this to be a collection of string values
representing numbers.
Previously the function would panic if the first element of a collection
was a non-number type, as we didn't attempt to convert it to a number
before calling the cty `Add` method.
This commit introduces a capsule type, `TypeType`, which is used to
extricate type information from the console-only `type` function. In
combination with the `TypeType` mark, this allows us to restrict the use
of this function to top-level display of a value's type. Any other use
of `type()` will result in an error diagnostic.
These instances of marks.Raw usage were semantically only testing the
properties of combining multiple marks. Testing this with an arbitrary
value for the mark is just as valid and clearer.
