Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meld consecutive template string literals #584

Merged
merged 1 commit into from Jan 30, 2023

Conversation

alisdair
Copy link
Member

When processing a template string, the lexer can emit multiple string literal tokens for what ought to be a single string literal. This occurs when the string contains escape sequences, or consecutive characters which are indistinguishable from escape sequences at tokenization time.

This leads to a confusing AST and causes heuristics about template expressions to fail. Specifically, when parsing a traversal with an index, a key value containing an escape symbol will cause the parser to generate an index expression instead of a traversal.

This commit adds a post-processing step to the template parser to meld any sequences of string literals into a single string literal. Existing tests covered the previous misbehaviour (several of which had comments apologizing for it), and have been updated accordingly.

The new behaviour of the IsStringLiteral method of TemplateExpr is covered with a new set of tests.


This change fixes a downstream bug in Terraform which is not currently tracked on GitHub. Apply the following configuration:

resource "null_resource" "none" {
}

Then update the configuration as follows:

resource "null_resource" "none" {
  for_each = toset(["$foo"])
}

moved {
  from = null_resource.none
  to   = null_resource.none["$foo"]
}

This should plan the move operation. Without this change, it results in the following error:

╷
│ Error: Invalid expression
│
│   on main.tf line 7, in moved:
│    7:   to   = null_resource.none["$foo"]
│
│ A single static variable reference is required: only attribute access and
│ indexing with constant keys. No calculations, function calls, template
│ expressions, etc are allowed here.
╵

@alisdair alisdair self-assigned this Jan 27, 2023
@alisdair alisdair requested review from a team and removed request for hc-github-team-tf-core January 27, 2023 20:36
hclsyntax/expression_template_test.go Outdated Show resolved Hide resolved
When processing a template string, the lexer can emit multiple string
literal tokens for what ought to be a single string literal. This occurs
when the string contains escape sequences, or consecutive characters
which are indistinguishable from escape sequences at tokenization time.

This leads to a confusing AST and causes heuristics about template
expressions to fail. Specifically, when parsing a traversal with an
index, a key value containing an escape symbol will cause the parser to
generate an index expression instead of a traversal.

This commit adds a post-processing step to the template parser to meld
any sequences of string literals into a single string literal. Existing
tests covered the previous misbehaviour (several of which had comments
apologizing for it), and have been updated accordingly.

The new behaviour of the `IsStringLiteral` method of `TemplateExpr` is
covered with a new set of tests.
@alisdair alisdair force-pushed the alisdair/meld-consecutive-string-literals branch from 81f4cda to 5fe5697 Compare January 27, 2023 21:00
@alisdair alisdair merged commit 8849dbc into main Jan 30, 2023
@alisdair alisdair deleted the alisdair/meld-consecutive-string-literals branch January 30, 2023 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants