Allow for escaping of variable syntax in serverless.yml

[brianneisler]

Allow for escaping of reserved variable characters (i.e. ${...)?

From this forum post http://forum.serverless.com/t/escaping-variable-syntax/1045?u=brianneisler

So suppose I would like to use my/topic/something/${timestamp()} as an S3 object key for writing to an S3 bucket. ${timestamp()} is a valid CloudFormation syntax for referencing IoT SQL functions. However, since this syntax is placed in Serverless file, the framework will look at it as if I'm referencing Serverless timestamp() variable and will obviously return an error below:

Invalid variable reference syntax for variable timestamp(). You can only reference env vars, options, & files. You can check our docs for more info.
I tried escaping it with a backslash, but it seems that escaping is then itself escaped as my backslashes are propagated further to the CloudFormation.

We should allow for this type of character escape.

Perhaps a simple approach is \${}

Proposed solution

(Added in later turn by maintainers)

Ideally \${} and \\\${} should escape, but \\${} and \\\\${} not. That can neatly be achieved by resolving templates via state machine (as one here: https://github.com/medikoo/es6-template-strings/blob/6a7ec2ec4fdf95176f48584e550a81c494c03b1c/compile.js), which should also be intelligent in handling nested variable resolution properly.

1. Implement a state machine, through which from every string we can retrieve eventual configured variables
2. Apply a variablesSynteaxregex on resolved variable, to ensure it matches our format, and if it passes resolve further variable as it happens currently

[pmuens]

Interesting! We have the possibility to define an own variableSyntax via provider.variableSyntax.

Here's an example on how to use it: #3561 (comment)

Could this potentially be used to mitigate this problem?

[brianneisler]

@pmuens, i think it can but as the forum post points out that's a bit extreme for something so simple. Seems convenient to offer a simple escape mechanism.

[TrentBartlem]

Yeah, I kinda need to escape ${cognito-identity.amazonaws.com:sub} and pass it through to cloudformation intact, which is currently not possible.

[TrentBartlem]

Note that the workaround (variableSyntax: "\\${{([\\s\\S]+?)}}") works well enough; however it means all the Serverless variables have to change to something like ${{env:SOMETHING}} throughout the entire file.

[walery]

Note that this workaround didn't work if you resolve nested variables like this:

${{self:custom.${{self:provider.stage}}.var}}

To be able to do this you need to use this variableSyntax:

variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}"

This is copied from official documentation (https://serverless.com/framework/docs/providers/aws/guide/variables#using-custom-variable-syntax).
Just want to share it here because it took me some time to find out.

And because I'm already writing comment: Escaping like this \${} would be really nice!

[ernieay]

another workaround would be to escape the ${ part by merging separate parts using the Join function:

Key:
Fn::Join: ['', ['$', '{timestamp()}']]

[mhama]

+1

As I set partition key for IoT Topic Rule (Kinesis Action), I want to put "${topic()}" value.

PartitionKey: "${topic}"

does't work (as expected).

So I used ernieay's workaround.

PartitionKey:
                  Fn::Join: ['', ['$', '{topic()}']]

But it could be simpler.

[cornfeedhobo]

Maybe escaping should follow the same pattern as terraform; provide a double dollar sign ($${}) option to escape variables that the user expects CloudFormation to parse, and not Serverless?

You can escape interpolation with double dollar signs: $${foo} will be rendered as a literal ${foo}.

Reference: https://www.terraform.io/docs/configuration/interpolation.html

[kbsanders]

For my use case, I need to be able to escape curly brackets that appear in paths when importing files via ${file()}.

This breaks...

functions:
  getGroup: '${file(./endpoints/Groups/{groupId}/GET/serverless.yml)}'

Ideally I would like to use something like backslash to escape the special characters...

functions:
  getGroup: '${file(./endpoints/Groups/\{groupId\}/GET/serverless.yml)}'

[jamesrusso]

I too am trying to include ${aws:username} in a resource definition for a s3 policy in order to permit access to s3 buckets which match the username.

It would be nice to be able to pass a literal without interpolation.

[FreddyJD]

@kbsanders any luck ?

For my use case, I need to be able to escape curly brackets that appear in paths when importing files via ${file()}.

This breaks...

functions:
  getGroup: '${file(./endpoints/Groups/{groupId}/GET/serverless.yml)}'

Ideally I would like to use something like backslash to escape the special characters...

functions:
  getGroup: '${file(./endpoints/Groups/\{groupId\}/GET/serverless.yml)}'

[kbsanders]

@FreddyJD No, I ended up using a different character (underscores) to represent path variables.

functions:
  getGroup: '${file(./endpoints/Groups/_groupId_/GET/serverless.yml)}'

It's been a while since I've worked on the project where this code is used though, so I don't know if any improvements have been made to allow escaping curly brackets or not in serverless config files.

[skrosoft]

Any update on this? I need it too +1

[pmyjavec]

I just hit this myself, it's a frustrating issue for people who need to pass a literal into cloud formation configuration.

[trobert2]

Passing literals into cloud formation is something that you do quite often.
Is there a cleaner way than the proposal to use Join?

[aj-bagwell]

I use the serverless-cloudformation-sub-variables plugin then you can use things like#{!timestamp()} or #{!iot:Certificate.Subject.CommonName} to get the apropriate template through.

The plugin converts arn:aws:iot:${self:provider.region}:#{AWS::AccountId}:some/topic/#{!iot:Certificate.Subject.CommonName}/# to {"Fn::Sub": "arn:aws:iot:eu-west-1:${AWS::AccountId}:some/topic/${!iot:Certificate.Subject.CommonName}/#"} then cloudformation Sub function treats substitutions starting with ! as a literal so you end up with arn:aws:iot:us-east-1:12345678:some/topic/${iot:Certificate.Subject.CommonName}/#

[medikoo]

I believe main use case was to address collision with AWS variables syntax (#3184), and that was already solved through improved regular expression.

Still I believe for completeness there should be some escape mechanism respected, I've outlined in main description a proposal on how we can achieve that

[medikoo]

Closing, as escaping is supported by a new resolver, e.g \${self:} will be taken literally

[dhruvsaxena1998]

I've a variable ${env:MYSQL_DB_PASSWORD} that is being set from gitlab ci/cd. This MYSQL_DB_PASSWORD resolved in string which has $ in it. And this is breaking the value.

I've tried using \${env:MYSQL_DB_PASSWORD} but it's just printing it as string not getting it's variable. Any help will be very appreciated.

[medikoo]

@dhruvsaxena1998 if you feel that escape mechanism doesn't work, firstly please ensure you're on latest version of a Framework, and if it confirms to be the case there, please open a bug report, documenting a minimal reproducible case that doesn't involve plugins.

[muraliavarma]

I have noticed that I had to sometimes double escape with \\${env:...} if used inside a string value.