| Age | Commit message (Collapse) | Author |
|---|
|
(commonly occurs in editors for completions)
Detect value expressions of the ExprSyntaxError type when parsing object constructor expressions and use them to add an item to the result even though we skip parsing the object due to recovery after the invalid expression.
This allows the Terraform language server to support completions for object attributes after a dot was typed.
|
|
We were calling .Range() on any unknown sourceVal, without first checking
whether it was marked. That method panics if called on a marked value,
so we need to strip that off first.
While testing this I found some return paths that weren't properly
transferring the source value's marks to the output, and so this also
addresses those so that all return paths preserve whatever markings are
present on the source value.
In particular, if a non-list/set/tuple value gets "upgraded" into a tuple
then we must transfer its marks onto the tuple, because the decision about
constructing that value was based on characteristics of the source value.
|
|
* Add additional function for parsing traversals with [*] keys
* add more context around skipped test cases
|
|
Due to the quite messy heritage of this codebase -- including a large part
of it being just a fork of my earlier personal project ZCL -- there were
many different conventions for how to pretty-print and diff values in the
tests in different parts of the codebase.
To reduce the dependency sprawl, this commit now standardizes on:
- github.com/davecgh/go-spew for pretty-printing
- github.com/google/go-cmp for diffing
These two dependencies were already present anyway, are the most general
out of all of the candidates, and are also already in use by at least some
of HCL's most significant callers, such as HashiCorp Terraform.
The version of go-cmp we were previously using seems to have a bug that
causes the tests to crash when run under the Go race detector, so I've
also upgraded that dependency to latest here to clear that bug.
|
|
|
|
|
|
|
|
for namespaced functions
|
|
|
|
|
|
Typo fix in spec.md
|
|
Now that we have namespaced functions, and implementations like
Terraform can add functions based on configuration, the reason for an
unknown function call name becomes a little less clear. Because
functions are populated outside of the hcl package scope, there isn't
enough context to provide a useful diagnostic to the user.
We can create a new Diagnostic.Extra value for
FunctionCallUnknownDiagExtra to indicate specifically when a diagnostic
is created due to an unknown function name. This will carry back the
namespace and function name for the caller to inspect, which will allow
refinement of the diagnostic based on information only known to the
caller.
|
|
|
|
|
|
Resolves #650
|
|
|
|
This introduces a new syntax which allows function names to have namespace
prefixes, with the different name parts separated by a double-colon "::"
as is common in various other C-derived languages which need to
distinguish between scope resolution and attribute/field traversal.
Because HCL has separate namespaces for functions and variables, we need
to use different punctuation for each to avoid creating parsing ambiguity
that could be resolved only with infinite lookahead.
We cannot retroactively change the representation of function names to be
a slice of names without breaking the existing API, and so we instead
adopt a convention of packing the multi-part names into single strings
which the parser guarantees will always be a series of valid identifiers
separated by the literal "::" sequence. That means that applications will
make namespaced functions available in the EvalContext by naming them in
a way that matches this convention.
This is still a subtle compatibility break for any implementation of the
syntax-agnostic HCL API against another syntax, because it may now
encounter function names in the function table that are not entirely
valid identifiers. However, that's okay in practice because a calling
application is always in full control of both which syntaxes it supports
and which functions it places in the function table, and so an application
using some other syntax can simply avoid using namespaced functions until
that syntax is updated to understand the new convention.
This initial commit only includes the basic functionality and does not yet
update the specification or specification test suite. It also has only
minimal unit tests of the parser and evaluator. Before finalizing this
in a release we would need to complete that work to make sure everything
is consistent and that we have sufficient regression tests for this new
capability.
|
|
In the modern Go Modules-based toolchain we can avoid the need to globally
install this tool first by running it this way. As a bonus, the toolchain
will also install the version of the module we have specified in go.mod,
thereby locking us in to a particular version until we intentionally
upgrade.
The other third-party generator tools we use here aren't written in Go and
so we can't do the same for those right now, but maybe we'll find a nicer
way to handle those later too.
|
|
Correct mark handling for some conditional values.
Find correct refinement for overlapping ranges which could not have been
compared with `GreaterThan`. Also map inclusive flags for numeric
ranges.
Correct handling of DefinitelyNotNull collections.
Return a known null early when both conditional branches are null.
|
|
|
|
When attempting to determine the final length range for a conditional
expression with collections, the length values may still be unknown.
Always use `Range()` to get the lower and upper bounds.
|
|
The interactions between value marks and unknown value refinements can be
a little tricky, so this pair of new tests cover two examples of that
interaction that are currently working and ought to stay that way.
|
|
Signed-off-by: Jakub Martin <kubam@spacelift.io>
|
|
To match with the Unicode support in Go 1.21, we'll now use the Unicode 15
tables when we're normalizing Unicode strings and when counting
user-perceived characters ("grapheme clusters") for source position
purposes.
|
|
There is no limit to the length of string prefixes produced by template
expressions, so in rare cases they may return a refined unknown string
has too long a prefix.
The cty's msgpack decoder limits the size of an acceptable refinements
to 1 kiB, so such a value cannot be handled and an error occurs.
This change limits the length of prefixes to 128 B, so overly long
prefixes are no longer an issue in most cases.
|
|
We know that a splat expression can never produce a null result, and also
in many cases we can use length refinements from the source collection to
also refine the destination collection because we know that a splat
expression produces exactly one result for each input element.
This also allows us to be a little more precise in the case where the
splat operator is projecting a non-list/set value into a zero or one
element list and we know the source value isn't null. This refinement is
a bit more marginal since it would be weird to apply the splat operator
to a value already known to be non-null anyway, but the refinement might
come from far away from the splat expression and so could still have
useful downstream effects in some cases.
|
|
When ConditionalExpr has an unknown predicate it can still often infer
some refinement to the range of its result by noticing characteristics
that the two results have in common.
In all cases we can test if either result could be null and return a
definitely-not-null unknown value if not.
For two known numbers we can constrain the range to be between those two
numbers. This is primarily aimed at the common case where the two possible
results are zero and one, which significantly constrains the range.
For two known collections of the same kind we can constrain the length
to be between the two collection lengths.
In these last two cases we can also sometimes collapse the unknown into
a known value if the range gets reduced enough. For example, if choosing
between two collections of the same length we might return a known
collection of that length containing unknown elements, rather than an
unknown collection.
|
|
If we encounter an interpolated unknown value during template rendering,
we can report the partial buffer we've completed so far as the refined
prefix of the resulting unknown value, which can then potentially allow
downstream comparisons to produce a known false result instead of unknown
if the prefix is sufficient to satisfy them.
|
|
This new concept allows constraining the range of an unknown value beyond
what can be captured in a type constraint. We'll make more use of this
in subsequent commits.
|
|
* [COMPLIANCE] Add Copyright and License Headers
* add copywrite file and revert headers in testdata
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com>
|
|
|
|
Meld consecutive template string literals
|
|
Object elements can be separated by comma or newline
|
|
|
|
Update spec.md to avoid suggesting ! is a binary operator
|
|
When processing a template string, the lexer can emit multiple string
literal tokens for what ought to be a single string literal. This occurs
when the string contains escape sequences, or consecutive characters
which are indistinguishable from escape sequences at tokenization time.
This leads to a confusing AST and causes heuristics about template
expressions to fail. Specifically, when parsing a traversal with an
index, a key value containing an escape symbol will cause the parser to
generate an index expression instead of a traversal.
This commit adds a post-processing step to the template parser to meld
any sequences of string literals into a single string literal. Existing
tests covered the previous misbehaviour (several of which had comments
apologizing for it), and have been updated accordingly.
The new behaviour of the `IsStringLiteral` method of `TemplateExpr` is
covered with a new set of tests.
|
|
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
|
|
Update go-cty and improve documentation for optional and default attributes
|
|
|
|
|
|
|
|
"within within" should be "within"
|
|
|
|
The current spec appears to suggest that the bang is a binary operator, which would enable
expressions such as `foo ! bar`.
https://github.com/hashicorp/hcl/blob/main/hclsyntax/parser.go#L1070 seems to suggest this is only
a unary operator.
|
|
The primary goal of the diagnostics design in HCL is to return
high-quality diagnostics messages primarily for human consumption, and so
their regular structure is only machine-processable in a general sense
where we treat all diagnostics as subject to the same processing.
A few times now we've ended up wanting to carry some additional optional
contextual information along with the diagnostic, for example so that a
more advanced diagnostics renderer might optionally annotate a diagnostic
with extra notes to help the reader debug.
We got pretty far with our previous extension of hcl.Diagnostic to include
the Expression and EvalContext fields, which allow an advanced diagnostic
renderer to offer hints about what values contributed to the expression
that failed, but some context is even more specific than that, or is
defined by the application itself and therefore not appropriate to model
directly here in HCL.
As a pragmatic compromise then, here we introduce one more field Extra
to hcl.Diagnostic, which comes with a documented convention of placing
into it situation-specific values that implement particular interfaces,
and therefore a diagnostics renderer or other consumer can potentially
"sniff" this field for particular interfaces it knows about and treat them
in a special way if present.
Since there is only one field here that might end up being asked to
capture multiple extra values as the call stack unwinds, there is also a
simple predefined protocol for "unwrapping" extra values in order to find
nested implementations within.
For callers that are prepared to require Go 1.18, the helper function
hcl.DiagnosticExtra provides a type-assertion-like mechanism for sniffing
for a particular interface type while automatically respecting the nesting
protocol. For the moment that function lives behind a build constraint
so that callers which are not yet ready to use Go 1.18 can continue to
use other parts of HCL, and can implement a non-generic equivalent of
this function within their own codebase if absolutely necessary.
As an initial example to demonstrate the idea I've also implemented some
extra information for error diagnostics returned from FunctionCallExpr,
which gives the name of the function being called and, if the diagnostic
is describing an error returned by the function itself, a direct reference
to the raw error value returned from the function call. I anticipate a
diagnostic renderer sniffing for hclsyntax.FunctionCallDiagExtra to see
if a particular diagnostic is related to a function call, and if so to
include additional context about the signature of that function in the
diagnostic messages (by correlating with the function in the EvalContext
functions table). For example:
While calling: join(separator, list)
An example application-specific "extra value" could be for Terraform to
annotate diagnostics that relate to situations where an unknown value is
invalid, or where a "sensitive" value (a Terraform-specific value mark) is
invalid, so that the diagnostic renderer can avoid distracting users with
"red herring" commentary about unknown or sensitive values unless they
seem likely to be relevant to the error being printed.
|
|
Port fuzz testing to Go 1.18 native fuzzing
|
|
|
|
|
|
For a long time now we've had a very simplistic error message for the
case of conditional expression result arms not having the same type, which
only works for situations where the two types have differing "friendly
names" down in the cty layer.
Unfortunately due to the typical complexity of the structural type kinds
(object and tuple types) their friendly names are just "object" and
"tuple", which tends to lead us to seemingly-incorrect error messages
like:
The true and false result expressions must have consistent types.
The given expressions are object and object, respectively.
This then is an attempt to use some more specialized messaging in some
of the situations that led to that sort of weird message before. In
particular, this handles:
- both types are object types but their attributes don't match
- both types are tuple types but their elements don't match
- both types are the same kind of collection of either object or tuple
types which don't match
These are the three _shallow_ cases that the previous logic wasn't able to
properly describe. This still leaves unaddressed a hopefully-less-common
case of nested collections with differing structural types in their
depths, but still avoids generating a confusing error message by instead
generating a _very vague but still correct_ error message:
At least one deeply-nested attribute or element is not compatible
across both the 'true' and the 'false' value.
My intent here is to make HCL return something precise enough _most of the
time_, without letting perfect be the enemy of the good. This will
generate some quite obnoxious long messages for particularly complex
nested structures, but so far it appears that such values are relatively
rare inside conditional expressions and so we'll wait to see what arises
in practice before trying to handle those situations more concisely.
Ideally I would like to include some actionable feedback that in some
cases it can help to explicitly convert ambiguously-typed expressions
like "null" or tuples intended to be lists to the intended type, so that
the type unification step has more information to infer the author intent.
However, HCL itself doesn't have any builtins for such conversions and so
today any messaging about that would need to be generated up at the
application layer so the application can refer to whatever functions/etc
it provides for type conversion. It isn't clear how to do that with the
current design, so we'll leave that to be addressed another day.
|
|
|
