AWS Lambda Runtime Upgrade Suggestions

[chezsmithy]

Type of discussion.

I'm proposing an idea

Tell us more.

Curious if anyone else might be interested in having AWS Lambda runtime upgrade support?

We currently allow our developers to request the runtime in custom app-spec files we process in our build pipelines. As such we would need a regex manager with a source of runtime reference data in renovate to support suggesting automated upgrades.

Example YML from our app-spec:

lambda_functions:
        - function_name: s3_events
          handler: s3events
          runtime: go1.x

[viceice]

please add an example about what should be updated in detail. i don't see something in your sample. go1.x doesn't look like a updatable version.

[chezsmithy]

The runtime list can be found here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

[chezsmithy]

For example if our app spec contained:

runtime: python3.9

We would expect an update to:

runtime: python3.10

[viceice]

The runtime list can be found here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

ideally there exists a json or yaml file or API to query. html parsing isn't very safe.

[rarkins]

What is the "app-spec"? Is there public documentation for it, and is there a public list of available values for the runtimes?

[chezsmithy]

App-spec is custom yaml used in our CICD pipeline so developer can specify the runtime. I'd imagine other users of renovate would have something similar in their pipelines to inform the pipeline which lambda image to package their server-less source into.

[chezsmithy]

There doesn't appear to be a public source I can find which is "data" vs the web source I posted above. I'll do some more hunting and ask our AWS technical account manager.

[chezsmithy]

Also adding that the runtimes evolve really slowly. Maybe a release a year per language.

[rarkins]

If they're slow then we could maybe have a manually curated datasource for it.

The next question is whether any of the file formats are standard or whether you need to craft your own regex manager

[chezsmithy]

It's going to likely be regex in many cases. However for folks using the AWS serverless spec (https://www.serverless.com/framework/docs/providers/aws/guide/serverless.yml) they could like get auto updates of provider: -> runtime:

[chezsmithy]

@secustor per your kind comments on the PR I opened I wanted to continue the dialog regarding AWS Lambda updates here.

As you review the previous comments in this discussion we noted that a custom manager can pretty easily handle the needed updates, and I proved that in this sample repo: https://github.com/chezsmithy/renovate-aws-lambda

However previously we noted that the largest issue was having a well maintained source of AWS lambda runtimes.

Thanks to the good work on the endoflife-date datasource, I identified that the endoflife-date API does have an aws-lambda dataset.

However, upon further investigation I noted a problem with using the endoflife-date datasource out of the box. The endoflife-date datasoruce expects that the packageName passed in maps to the specific json dataset maintained by endoflife vs the packageName contained in that json.

For example, the eks dataset for example looks like this:

[
  {
    "cycle": "1.26",
    "releaseDate": "2023-04-11",
    "eol": false,
    "latest": "1.26-eks-1",
    "latestReleaseDate": "2023-04-11",
    "lts": false
  },
  {
    "cycle": "1.25",
    "releaseDate": "2023-02-21",
    "eol": "2024-05-01",
    "latest": "1.25-eks-3",
    "latestReleaseDate": "2023-04-06",
    "lts": false
  },
  {
    "cycle": "1.24",
    "eol": "2024-01-01",
    "latest": "1.24-eks-6",
    "latestReleaseDate": "2023-04-06",
    "releaseDate": "2022-11-15",
    "lts": false
  },
  {
    "cycle": "1.23",
    "eol": "2023-10-01",
    "latest": "1.23-eks-8",
    "releaseDate": "2022-08-11",
    "latestReleaseDate": "2023-04-06",
    "lts": false
  },
  {
    "cycle": "1.22",
    "eol": "2023-06-04",
    "latest": "1.22-eks-12",
    "releaseDate": "2022-04-04",
    "latestReleaseDate": "2023-04-06",
    "lts": false
  },
  {
    "cycle": "1.21",
    "eol": "2023-02-15",
    "latest": "1.21-eks-17",
    "releaseDate": "2021-07-19",
    "latestReleaseDate": "2023-04-06",
    "lts": false
  },
  {
    "cycle": "1.20",
    "eol": "2022-11-01",
    "latest": "1.20-eks-14",
    "releaseDate": "2021-05-18",
    "latestReleaseDate": "2023-04-06",
    "lts": false
  },
  {
    "cycle": "1.19",
    "eol": "2022-08-01",
    "latest": "1.19-eks-11",
    "releaseDate": "2021-02-16",
    "latestReleaseDate": "2022-08-15",
    "lts": false
  },
  {
    "cycle": "1.18",
    "eol": "2022-03-31",
    "latest": "1.18-eks-13",
    "releaseDate": "2020-10-13",
    "latestReleaseDate": "2022-08-15",
    "lts": false
  }
]

And contains only information about the aws-eks "package".

In the case of the aws-lambda data it looks like this:

[
    {
        "cycle": "python3.12",
        "releaseLabel": "Python 3.12",
        "releaseDate": "2023-12-14",
        "support": true,
        "eol": false,
        "link": "https://aws.amazon.com/about-aws/whats-new/2023/12/aws-lambda-support-python-3-12/",
        "latest": "python3.12",
        "latestReleaseDate": "2023-12-14",
        "lts": false
    },
    {
        "cycle": "nodejs20.x",
        "releaseLabel": "Node.js 20",
        "releaseDate": "2023-11-15",
        "support": true,
        "eol": false,
        "link": "https://aws.amazon.com/blogs/compute/node-js-20-x-runtime-now-available-in-aws-lambda/",
        "latest": "nodejs20.x",
        "latestReleaseDate": "2023-11-15",
        "lts": false
    },
    {
        "cycle": "python3.11",
        "releaseLabel": "Python 3.11",
        "releaseDate": "2023-07-27",
        "support": true,
        "eol": false,
        "link": "https://aws.amazon.com/blogs/compute/python-3-11-runtime-now-available-in-aws-lambda/",
        "latest": "python3.11",
        "latestReleaseDate": "2023-07-27",
        "lts": false
    },
    {
        "cycle": "python3.10",
        "releaseLabel": "Python 3.10",
        "releaseDate": "2023-04-18",
        "support": true,
        "eol": false,
        "link": "https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/",
        "latest": "python3.10",
        "latestReleaseDate": "2023-04-18",
        "lts": false
    }
]

And contains information about "many" aws-lambda runtime "packages".

As I investigated further, I noted that there wasn't a clean way to get the extracted packageName from the custom manager to the endoflife-date datasource as it's expected it's a single packageName.

My proposed solution was to clone the endoflife-date datasource and enhance it so that the endoflife dataset (aws-lambda) can be hard coded in the datasource for example as there isn't a logical way I could see to pass that down from the customer manager, and instead the packageName could become the extracted packageName from the custom manager.

Happy to discuss at length! Thanks.

[rarkins]

I'm not sure I understand. Is this so that you can have one datasource, but multiple transforms? To avoid duplicating similar custom datasources from same source?

[secustor]

Basically we ship custom datasource definitions with included presets.

[rarkins]

You've told me what, but not why :)

[secustor]

For UX purposes, as we have some variants used regularly and these can hard to find in discussions.

Adding all variants to the docs can be also cumbersome, if the alternaive is a extend: ['custom.datasources:all']

[rarkins]

Happy to use presets for this purpose

[AlexC]

I was able to get this working with a customManager and endoflife-date for our Terraform files:

packageRules: [
    {
      matchDatasources: ['endoflife-date'],
      matchPackageNames: ['aws-lambda'],
      extractVersion: '^nodejs(?<version>[0-9]{2})\\.x$',
    },
],
customManagers: [
    {
      customType: 'regex',
      description: 'Update AWS Lambda NodeJS runtime in TF files',
      fileMatch: ['.+\\.tf$'],
      matchStrings: [
        '#\\s*renovate:\\s*datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\s*runtime\\s*=\\s*"nodejs(?<currentValue>[0-9]{2})\\.x"',
      ],
      versioningTemplate: '{{#if versioning}}{{{versioning}}}{{/if}}',
    },
]

Example:

resource "aws_lambda_function" "foobar" {
  ...
  # renovate: datasource=endoflife-date depName=aws-lambda versioning=loose
  runtime          = "nodejs12.x"
  ...
}