Preliminary work to compile on Amazon Linux 2023 #122

mnapoli · 2023-09-18T12:14:32Z

AL2023 is eventually going to happen. Out of curiosity, I drafted this PR trying to build our layers for AL2023.

It removes a lot of custom system libraries we compile because AL2023 contains more recent versions: https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.1.html

We'll need to reevaluate when we actually get AL2023 Lambda images (right now we have the generic AL2023, not specific to Lambda), but it is looking good.

I'm opening this PR to gather early feedback and see if I'm missing something.

mnapoli · 2023-09-18T12:15:03Z

php-80/Dockerfile

-# - libzip: minimum required CMAKE version 3.0.
-RUN LD_LIBRARY_PATH= yum install -y cmake3
-# Override the default `cmake`
-RUN ln -s /usr/bin/cmake3 /usr/bin/cmake


cmake3 is installed by default

This is due to AL2 having both cmake and cmake3 packages to ensure that software needing cmake that didn't yet work with cmake3 could continue to be built.

For AL2023, it's all cmake version 3.

mnapoli · 2023-09-18T12:15:51Z

php-81/Dockerfile

-#   - curl
-#   - php
-RUN yum install -y perl-IPC-Cmd
-ENV VERSION_OPENSSL=3.0.10


Native version: 3.0.8

Security issues in the native version?

I tried asking around but have no answers yet. Do you know if the versions compiled in AL might have some security fixes applied?

I'm just surprised AL would ship with security issues, how do all customers do? I wouldn't expect everyone to compile everything from source, right?

We backport security fixes to the OpenSSL versions that we ship. See https://alas.aws.amazon.com/alas2023.html for updates.

For example: https://alas.aws.amazon.com/AL2023/ALAS-2023-306.html and https://alas.aws.amazon.com/AL2023/ALAS-2023-222.html are some recent ones.

If there's anything from newer OpenSSL that you need, please do let us know and submit an issue to https://github.com/amazonlinux/amazon-linux-2023/ for us to look at.

We need a version that supports quic possibly, for http/3 in curl, also. I am not sure how long it takes AWS to apply security fixes. I like the flexibility that keeping this gives us to easily re-compile, or for me to fork and re-compile to get something patched immediately.

mnapoli · 2023-09-18T12:16:12Z

php-81/Dockerfile

-# Needed by:
-#   - php
-#   - libnghttp2
-ENV VERSION_XML2=2.11.5


Native version: 2.10.4

Security issues in the native version?

We do backport security fixes to the version we ship in Amazon Linux. See https://alas.aws.amazon.com/AL2023/ALAS-2023-343.html for an example of this for libxml2.

mnapoli · 2023-09-18T12:16:30Z

php-81/Dockerfile

-#   - OpenSSL
-# Needed by:
-#   - curl
-ENV VERSION_LIBSSH2=1.11.0


Native version: 1.10.0

Security issues in the native version?

mnapoli · 2023-09-18T12:16:49Z

php-81/Dockerfile

-#   - libxml2
-# Needed by:
-#   - curl
-ENV VERSION_NGHTTP2=1.56.0


Native version: 1.55.1

mnapoli · 2023-09-18T12:17:08Z

php-81/Dockerfile

-# #   - libnghttp2
-# # Needed by:
-# #   - php
-ENV VERSION_CURL=8.3.0


Native version: 8.2.1

This won't be possible, once we need to start building curl with http/3 enabled.

Note that we do move curl forward in Amazon Linux fairly regularly, often as part of patching security issues.

If there's anything specifically not enabled in our curl build that you need, please file an issue over at https://github.com/amazonlinux/amazon-linux-2023/issues so we can look into it.

Note that in AL2023 there is both a curl and curl-minimal package (and the associated libcurl and libcurl-minimal). The minimal ones are installed by default, and we try and capture the 99% of use cases there (e.g. don't build in ldap:// support and the like).

mnapoli · 2023-09-18T12:17:24Z

php-81/Dockerfile

-# https://github.com/nih-at/libzip/releases
-# Needed by:
-#   - php
-ENV VERSION_ZIP=1.10.1


Native version: 1.7.3

This native version is wayyyyy too old.

😄 thanks for all the feedback, happy I opened this this is useful (to know we may not want to drop all the compilation we are doing)

Could you open an issue up at https://github.com/amazonlinux/amazon-linux-2023/issues detailing the issues with the older version that would require an update?

mnapoli · 2023-09-18T12:18:25Z

php-81/Dockerfile

-#   - OpenSSL
-# Needed by:
-#   - php
-ENV VERSION_POSTGRES=15.4


Native version: 15.0

This is over a year old.

Could you cut us an issue at https://github.com/amazonlinux/amazon-linux-2023/issues detailing what from the client side you need updated?

mnapoli · 2023-09-18T12:18:40Z

php-81/Dockerfile

-# https://github.com/kkos/oniguruma/releases
-# Needed by:
-#   - php mbstring
-ENV VERSION_ONIG=6.9.8


Native version: 6.9.7.1

GrahamCampbell · 2023-09-18T12:49:12Z

AL2023 is eventually going to happen

I am actually not so sure. 😆

mnapoli · 2023-09-27T14:42:33Z

@GrahamCampbell that wasn't long :) aws/aws-lambda-base-images#92

GrahamCampbell · 2023-09-27T14:53:10Z

Ha, it's not available on the Lambda APIs and cloudformation yet. ;)

stewartsmith · 2023-10-09T18:05:47Z

php-80/Dockerfile

-        -DCMAKE_BUILD_TYPE=RELEASE
-RUN cmake  --build . --target install
-
-
 ###############################################################################
 # LIBSODIUM
 # https://github.com/jedisct1/libsodium/releases


Building libsodium shouldn't be required on AL2023 as there's a version in the repositories as of AL2023.2 (see https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.2.20230920.html )

Again, it is not the latest version.

php-81/Dockerfile

mnapoli · 2023-10-10T08:56:45Z

Thanks so much for jumping in the discussion @stewartsmith.

There is some balance that we need to find.

Would it be reasonable to apply the following logic:

When we need specific features (random example: HTTP3 support in curl) that are not present in AL2023, we recompile the libraries
The rest of the time, we rely on AL2023 versions for CVEs and security fixes

Is that reasonable enough? Can we assume that all CVEs are patched in AL2023 eventually? (I understand it may take a few hours/days depending on the issue)

The benefits of that approach would be:

our compilation scripts are simpler
our layers might potentially be lighter (faster cold starts)
security fixes are delivered in the Lambda OS, not via a new version of the Bref layers -> for most Bref users, that means that security fixes are actually applied faster, because most Bref users don't necessarily update Bref layers every week or so (meaning they are actually using older unpatched system libraries provided by Bref).

WDYT?

stewartsmith · 2023-10-12T14:50:52Z

Building anything from source and replacing an OS component with it carries various risks. Naturally it means you have to keep it updated, but there's also the nuance of building it with the same hardening and optimisations as the one in the OS, and ensuring ABI compatibility with the one in the OS. It's not something we recommend doing.

It is (of course) something which is possible to do, as it's basically what we do all the time when building updates for the OS. From a "does the Amazon Linux team support this" standpoint, replacing core system libraries falls into "can you reproduce the issue without that change" territory - i.e. a freedom that Free and Open Source Software gives you, but the vendor doesn't necessarily provide support for if you pick up the (metaphorical or literal) phone.

I would recommend building RPMs instead of manually from source tarballs though, as that way you capture all the hardening and optimization flags that are added in the RPM macros.

In Amazon Linux, security is job zero. We do fix security issues of all severity, and aim to do so not just in the most recent major version of the OS. The places where don't patch something basically boils down to risk outweighing the benefit - such as Low CVEs that are a quite invasive fix that we can't reasonably bring in without creating other, more severe issues.

tests/test_2_extensions.php

mnapoli · 2023-10-17T14:23:01Z

Thanks a lot @stewartsmith for chiming in. I've taken some time to think about it, and also discuss this with a few Bref users. For example I discussed this topic with a bank running on Bref (i.e. they have huge security constraints).

The feedback was quite simple: if we build our own versions, we (Bref project) are responsible for the security related to those libraries. If we don't, AWS is.

To summarize:

Bref users looking for strong security guarantees will usually prefer to delegate as much as possible to AWS (https://docs.aws.amazon.com/whitepapers/latest/security-overview-aws-lambda/the-shared-responsibility-model.html). This allows ticking a lot of boxes.
Additionally, most Bref users will not update the runtime versions frequently enough, leaving them more exposed to CVEs and security issues (in the case where we build libraries). By using system libraries instead, AWS is able to patch the underlying runtime automatically, which gives a more secure environment.

That leads me to conclude that we should aim to compile as few libraries as possible. This will benefit both extremes of the spectrum: users with strong security requirements and those that don't update frequently (i.e. care less about security, if I can say so).

The downside is that we have no way to control the frequency/speed at which AWS patches security issues. This basically means putting more trust into AWS over Bref regarding security, and I think that is a reasonable choice 😅

This PR is still WIP, but I'll be slowly updating it to reflect that. I.e. the build scripts should be simpler with AL2023 than AL2.

Also since we'll want to support both, I'll probably reorganize the code entirely. We will also likely not support PHP 8.0 with AL2023.

GrahamCampbell · 2023-10-17T22:34:05Z

php-80/Dockerfile

@@ -11,23 +11,14 @@ ARG VERSION_PHP=8.0.30
 # https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
 # AWS provides a Docker image that we use here:
 # https://github.com/amazonlinux/container-images/tree/amzn2
-FROM public.ecr.aws/lambda/provided:al2-${IMAGE_VERSION_SUFFIX} as build-environment
+FROM public.ecr.aws/lambda/provided:al2023-preview-${IMAGE_VERSION_SUFFIX} as build-environment


is there a way to pin this to al2023.2, rather than just al2023?

hmmm, not really. just al2023-preview.2023.10.17.17

shiki-sou · 2024-05-15T03:05:32Z

Hello,
Thank you for your continuous work on the Bref project.
It seems this ticket has been postponed for a little bit while.
I'm interested in how long Amazon Linux 2023 will be supported.
And is there any key milestones or timelines related to this support?

hrzbrg · 2024-05-15T05:13:35Z

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc.
See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

shiki-sou · 2024-05-15T06:14:43Z

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc. See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

Thank you for answering my question.
Sorry about not asking question clearly.

I want to know the schedual that brefphp/aws-lambda-layers support Amazon Linux 2023 which this ticket was working on (If I'm not misunderstanding).
Because I found that that brefphp/aws-lambda-layers only suppot amazon linux 2(provided.al2).
See the capture below (I get this capture from Lambda function which I deployed)

mnapoli · 2024-05-17T12:08:39Z

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

shiki-sou · 2024-05-20T06:27:08Z

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

Thank you for your reply.

mnapoli commented Sep 18, 2023

View reviewed changes

php-81/Dockerfile Outdated

# - libxml2

# Needed by:

# - curl

ENV VERSION_NGHTTP2=1.56.0

Copy link

Member Author

mnapoli Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 1.55.1

mnapoli commented Sep 18, 2023

View reviewed changes

stewartsmith reviewed Oct 9, 2023

View reviewed changes

php-81/Dockerfile Outdated Show resolved Hide resolved

This was referenced Oct 11, 2023

[Package Update Request] - libzip 1.10.1 amazonlinux/amazon-linux-2023#506

Open

[Package Request] - oniguruma amazonlinux/amazon-linux-2023#507

Open

GrahamCampbell suggested changes Oct 17, 2023

View reviewed changes

tests/test_2_extensions.php Show resolved Hide resolved

GrahamCampbell reviewed Oct 17, 2023

View reviewed changes

wip

cd90255

mnapoli force-pushed the al2023 branch from 4a32a1c to cd90255 Compare May 22, 2024 12:38

Preliminary work to compile on Amazon Linux 2023 #122

Are you sure you want to change the base?

Preliminary work to compile on Amazon Linux 2023 #122

Conversation

mnapoli commented Sep 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamCampbell commented Sep 18, 2023

mnapoli commented Sep 27, 2023

GrahamCampbell commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnapoli commented Oct 10, 2023

stewartsmith commented Oct 12, 2023

mnapoli commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shiki-sou commented May 15, 2024

hrzbrg commented May 15, 2024

shiki-sou commented May 15, 2024

mnapoli commented May 17, 2024

shiki-sou commented May 20, 2024

mnapoli commented Sep 18, 2023 •

edited