Allow to escape variables format

[0ptional]

Source of a reported issue was that "${StepFunctionArn}" in CF template was attempted to be resolved by our variable resolver, when it was unexpected.

Proposed solution:

We probably should support escaping such case, so "${StepFunctionArn}" will not invoke variable resolver but just strip leading slash.

---

Original report:

Using custom variableSyntax, serverless can't resolve variable value if it contains another variable:

serverless.yml

service: test

provider:
  name: aws
  stage: dev
  variableSyntax: "\\${((env|self|opt|file|cf|s3)[:\\(][ :a-zA-Z0-9._,\\-\\/\\(\\)]*?)}"
  environment:
    VAR: ${self:custom.bucketName}
  
custom:
  bucketName: mytest-bucket-${self:provider.stage}

sls package .serverless/serverless-stage.json

Custom variable bucketName gets correctly resolved to mytest-bucket-dev. But environment variable in provider gets resolved to ${deep:0}. See serverless-stage.json:

{
  "service": {
    "service": "test",
    "serviceObject": {
      "name": "test"
    },
    "provider": {
      "stage": "dev",
      "variableSyntax": "\\${((env|self|opt|file|cf|s3)[:\\(][ :a-zA-Z0-9._,\\-\\/\\(\\)]*?)}",
      "name": "aws",
      "environment": {
        "VAR": "${deep:0}"
      },
      "region": "us-east-1",
      "versionFunctions": true,
      "compiledCloudFormationTemplate": {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "The AWS CloudFormation template for this Serverless application",
        "Resources": {
          "ServerlessDeploymentBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
              "BucketEncryption": {
                "ServerSideEncryptionConfiguration": [
                  {
                    "ServerSideEncryptionByDefault": {
                      "SSEAlgorithm": "AES256"
                    }
                  }
                ]
              }
            }
          },
          "ServerlessDeploymentBucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
              "Bucket": {
                "Ref": "ServerlessDeploymentBucket"
              },
              "PolicyDocument": {
                "Statement": [
                  {
                    "Action": "s3:*",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Resource": [
                      {
                        "Fn::Join": [
                          "",
                          [
                            "arn:",
                            {
                              "Ref": "AWS::Partition"
                            },
                            ":s3:::",
                            {
                              "Ref": "ServerlessDeploymentBucket"
                            },
                            "/*"
                          ]
                        ]
                      }
                    ],
                    "Condition": {
                      "Bool": {
                        "aws:SecureTransport": false
                      }
                    }
                  }
                ]
              }
            }
          }
        },
        "Outputs": {
          "ServerlessDeploymentBucketName": {
            "Value": {
              "Ref": "ServerlessDeploymentBucket"
            }
          }
        }
      },
      "coreCloudFormationTemplate": {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "The AWS CloudFormation template for this Serverless application",
        "Resources": {
          "ServerlessDeploymentBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
              "BucketEncryption": {
                "ServerSideEncryptionConfiguration": [
                  {
                    "ServerSideEncryptionByDefault": {
                      "SSEAlgorithm": "AES256"
                    }
                  }
                ]
              }
            }
          },
          "ServerlessDeploymentBucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
              "Bucket": {
                "Ref": "ServerlessDeploymentBucket"
              },
              "PolicyDocument": {
                "Statement": [
                  {
                    "Action": "s3:*",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Resource": [
                      {
                        "Fn::Join": [
                          "",
                          [
                            "arn:",
                            {
                              "Ref": "AWS::Partition"
                            },
                            ":s3:::",
                            {
                              "Ref": "ServerlessDeploymentBucket"
                            },
                            "/*"
                          ]
                        ]
                      }
                    ],
                    "Condition": {
                      "Bool": {
                        "aws:SecureTransport": false
                      }
                    }
                  }
                ]
              }
            }
          }
        },
        "Outputs": {
          "ServerlessDeploymentBucketName": {
            "Value": {
              "Ref": "ServerlessDeploymentBucket"
            }
          }
        }
      }
    },
    "custom": {
      "bucketName": "mytest-bucket-dev"
    },
    "pluginsData": {},
    "functions": {},
    "configValidationMode": "warn",
    "serviceFilename": "serverless.yml",
    "layers": {},
    "initialServerlessConfig": {
      "service": {
        "$ref": "$[\"service\"][\"serviceObject\"]"
      },
      "provider": {
        "$ref": "$[\"service\"][\"provider\"]"
      },
      "custom": {
        "$ref": "$[\"service\"][\"custom\"]"
      },
      "functions": {
        "$ref": "$[\"service\"][\"functions\"]"
      },
      "package": {
        "artifactDirectoryName": "serverless/test/dev/1598020300313-2020-08-21T14:31:40.313Z"
      },
      "layers": {
        "$ref": "$[\"service\"][\"layers\"]"
      },
      "configValidationMode": "warn"
    },
    "isDashboardMonitoringPreconfigured": false
  },
  "package": {
    "artifactDirectoryName": "serverless/test/dev/1598020300313-2020-08-21T14:31:40.313Z",
    "artifact": ""
  }
}

Installed version

Framework Core: 1.79.0
Plugin: 3.7.1
SDK: 2.3.1
Components: 2.34.6

[medikoo]

@0ptional thanks for report. Can you respect all bug template remarks, and point at which exactly point you observe ENV_VAR1 not resolved as expected?

[0ptional]

@medikoo Sorry! I updated the report with full serverless.yml and output for serverless-stage.json file.

[medikoo]

@optional thanks for update. It's caused by variableSyntax configuration in your config. Is there any specific reason you override that?

[0ptional]

@medikoo Yes, I need it because it otherwise conflicts with Variable Syntax of CloudFormation:

Type: AWS::CloudWatch::Dashboard
Properties:
    DashboardName: ${self:custom.dashboardName}
    DashboardBody: !Sub 
        - |
            {
                "widgets": [
                    {
                        "type": "metric",
                        "properties": {
                            "view": "singleValue",
                            "metrics": [
                                [ "AWS/States", "ExecutionTime", "StateMachineArn", "${StepFunctionArn}" ]
                            ],
                            "region": "eu-central-1",
                        }
                    },
        - StepFunctionArn: !Ref TestStepFunctionsStateMachine

If I run it without setting variableSyntax, I get Invalid variable reference syntax for variable StepFunctionArn. Not sure how else I could get that working without modifying variableSyntax.

[medikoo]

Problem is that internally variable resolution engine uses temporary deep variable scope, and as you do not recognize it in your regex, it is not resolved. If you add it to list of recognized scopes (in regex) it'll resolve as expected.

Anyway it feels to me that real problem is collision of our syntax with format you use in CloudFormation template. I wonder how to tackle that in best way.

We probably should support escaping such case, so "\${StepFunctionArn}" will not invoke variable resolver but just strip leading slash.

Let me keep this issue open as reference to that specific problem

[0ptional]

@medikoo Ah, perfect. That solved my problem. Thank you!

variableSyntax: "\\${((env|self|opt|file|cf|s3|deep)[:\\(][ :a-zA-Z0-9._,\\-\\/\\(\\)]*?)}"

[medikoo]

Closing in favor of #3184, which is about same thing and has already involved discussion

[medikoo]

Duplicate of #3184