feat: update docker images and add support for different architectures #1755

BenjaminBergerM · 2024-02-03T21:37:15Z

Description

Since https://github.com/lambci/docker-lambda lambda docker images are deprecated, serverless offline using docker was not working with many languages. So I changes de docker image source from lambci/docker-lambda to public.ecr.aws/lambda/{runtime}.

Also added support to diffrent architectures, now works with all runtimes and architectures that aws has a public image con ecr.

Also change docker configuration so it could be run with the new images.

Follow serverless/serverless#10797 (comment) this instructions.

Motivation and Context

I have many services running with Serverless and Golang, i use this plugin to run them locally and since 29 of January i want able to run muy code.

Related issues: #1754 #1750

How Has This Been Tested?

I run all test, and also imported the package to deferent serverless applications i have using docker.

Screenshots (if appropriate):

jomp16

It sorta works using java11 runtime after changing this import line, but still not 100% on first run.

On first run, calling some REST API, the runtime gives an error ✖ Failed to fetch from http://localhost:50257/2018-06-01/ping with Not Found

But when I request the same REST API, it works (and all subsequently requests)

jomp16 · 2024-02-05T13:31:36Z

src/lambda/handler-runner/docker-runner/DockerRuntime.js

@@ -0,0 +1,33 @@
+import { supportedRuntimesArquitecture } from "../../config/index.js"


It does not run because of this line, it should be changed to

import { supportedRuntimesArquitecture } from "../../../config/supportedRuntimes.js"

I just changed this, sorry for the issue.

BenjaminBergerM · 2024-02-05T13:57:07Z

It sorta works using java11 runtime after changing this import line, but still not 100% on first run.

On first run, calling some REST API, the runtime gives an error ✖ Failed to fetch from http://localhost:50257/2018-06-01/ping with Not Found

But when I request the same REST API, it works (and all subsequently requests)

On this aws docker images i could find ping endpoint, so i remove ping function.

manfioLP · 2024-02-05T23:52:27Z

apparently this would solve a lot of issues
#1754
#1750
other issues mentioned on serverless repos too

LeoNero · 2024-02-06T01:01:49Z

Tested with a project using golang, provided.al2, and an M1 Mac and it seems to work well. No errors found so far.

ncostamagna · 2024-02-06T03:17:04Z

I am testing in Go and it works!

dOrgJelli · 2024-02-06T22:27:01Z

Tested this using python3.10 functions on linux and works great, thank you for this fix!

BenjaminBergerM · 2024-02-09T00:13:14Z

@dnalborczyk i just fix the lint error, can run again the actions workflow?

src/lambda/__tests__/fixtures/Lambda/LambdaFunctionThatReturnsJSONObject-fixture.js

src/lambda/__tests__/fixtures/Lambda/LambdaFunctionThatReturnsNativeString-fixture.js

dnalborczyk · 2024-02-09T00:31:16Z

thank you for the PR @BenjaminBergerM !

I just de-activated the docker tests because of the failures. I suppose we should activate those again. could you do this in this PR? 0e883f7

dnalborczyk · 2024-02-09T00:35:01Z

src/lambda/handler-runner/docker-runner/DockerContainer.js

  async request(event) {
-    const url = `http://${this.#dockerOptions.host}:${this.#port}/2015-03-31/functions/${this.#functionKey}/invocations`
+    const url = `http://${this.#dockerOptions.host}:${this.#port}/2015-03-31/functions/function/invocations`


is ${this.#functionKey} not working here?

On the aws images, to invoke a function is no need to use the function key.

https://docs.aws.amazon.com/lambda/latest/dg/go-image.html#go-image-clients

Heres an example on the documentation.

src/config/supportedRuntimes.js

src/lambda/handler-runner/docker-runner/DockerRuntime.js

BenjaminBergerM · 2024-02-10T21:53:29Z

thank you for the PR @BenjaminBergerM !

I just de-activated the docker tests because of the failures. I suppose we should activate those again. could you do this in this PR? 0e883f7

Hi @dnalborczyk, thanks you for the review!

I rebase the branch and activated the tests you de-activate previously.

Waiting for the final review!

src/lambda/handler-runner/docker-runner/DockerRuntime.js

LuidmilaDezhkina · 2024-02-15T15:10:50Z

Hello everyone! I want to express my appreciation for your PR contributions. While deploying the code to the cloud seems to be successful, I've encountered an error locally: '/lambda-entrypoint.sh: line 14: /var/runtime/bootstrap: No such file or directory.' Interestingly, I can see that the zip file was unpacked, and for each invocation, I can locate it at a path similar to '../.serverless-offline/...../fe9fdf79-23ca-49d6-9d1c-162876a7960b/code/bootstrap.' Any insights on what might be causing this issue?

src/config/supportedRuntimes.js

DorianMazur · 2024-04-20T19:03:32Z

There are some issues on ubuntu:

× 19 Apr 2024 19:54:44,280 [INFO] (rapid) INIT START(type: on-demand, phase: init)
  19 Apr 2024 19:54:44,280 [INFO] (rapid) The extension's directory "/opt/extensions" does not exist, assuming no extensions to be loaded.
  19 Apr 2024 19:54:44,280 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
Warning:  Apr 2024 19:54:44,281 [WARNING] (rapid) First fatal error stored in appctx: Runtime.InvalidEntrypoint
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INIT RTDONE(status: error)
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INIT REPORT(durationMs: 0.416000)
Error: 19 Apr 2024 19:54:44,281 [ERROR] (rapid) Init failed error=fork/exec /var/runtime/bootstrap: no such file or directory InvokeID=
Warning:  Apr 2024 19:54:44,281 [WARNING] (rapid) Shutdown initiated: spindown
  19 Apr 2024 19:54:44,281 [INFO] (rapid) Waiting for runtime domain processes termination
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INIT START(type: on-demand, phase: invoke)
  19 Apr 2024 19:54:44,281 [INFO] (rapid) The extension's directory "/opt/extensions" does not exist, assuming no extensions to be loaded.
  19 Apr 2024 19:54:44,281 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
Warning:  Apr 2024 19:54:44,281 [WARNING] (rapid) First fatal error stored in appctx: Runtime.InvalidEntrypoint
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INIT RTDONE(status: error)
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INIT REPORT(durationMs: 0.298000)
  19 Apr 2024 19:54:44,281 [INFO] (rapid) INVOKE START(requestId: cd2d4f30-d5ad-4f38-bd05-bc9574b90d13)
Error: 19 Apr 2024 19:54:44,281 [ERROR] (rapid) Invoke failed error=fork/exec /var/runtime/bootstrap: no such file or directory InvokeID=cd2d4f30-d5ad-4f38-bd05-bc9574b90d13
Error: 19 Apr 2024 19:54:44,281 [ERROR] (rapid) Invoke DONE failed: Runtime.InvalidEntrypoint
Warning:  Apr 2024 19:54:44,281 [WARNING] (rapid) Reset initiated: ReleaseFail
  19 Apr 2024 19:54:44,281 [INFO] (rapid) Waiting for runtime domain processes termination

Maybe this issue is related, it's hard to say (I need to dig deeper) -> aws/aws-lambda-base-images#26

There is also another alternative to lambci/lambda images (but they are not official docker images) -> https://github.com/mLupine/docker-lambda
Someone created a PR with mLupine/docker-lambda images for a serverless repo -> serverless/serverless#12348

DorianMazur · 2024-04-23T20:33:25Z

Okay, it should be fixed now. I will wait for tests and probably merge it. Thank you @BenjaminBergerM

BenjaminBergerM force-pushed the update-docker-image-source branch 2 times, most recently from 1a3b0da to 7390daa Compare February 3, 2024 21:45

BenjaminBergerM changed the title ~~WIP: Update docker image source~~ Draft: Update docker image source Feb 3, 2024

BenjaminBergerM force-pushed the update-docker-image-source branch 2 times, most recently from 676f9fd to 8be74c3 Compare February 3, 2024 21:56

BenjaminBergerM changed the title ~~Draft: Update docker image source~~ Update docker image source Feb 3, 2024

jomp16 reviewed Feb 5, 2024

View reviewed changes

BenjaminBergerM changed the title ~~Update docker image source~~ Docker Support - update docker image source Feb 5, 2024

BenjaminBergerM requested a review from jomp16 February 5, 2024 23:55

ncostamagna approved these changes Feb 6, 2024

View reviewed changes