Add lld linker to the centos, alpine and crossbuild images #446

janvorli · 2021-05-05T00:43:30Z

We will be switching runtime build to using lld linker, so we need it in the images that we use to build it.

mthalman · 2021-05-05T00:53:03Z

src/alpine/3.9/amd64/Dockerfile

@@ -20,6 +20,7 @@ RUN apk add --no-cache \
        libtool \
        libunwind-dev \
        linux-headers \
+        lld \


Alpine 3.9 is EOL and will be deleted soon: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/442

We still use it to build runtime for Alpine and we don't have equivalent one for 3.10 yet. Would you like me to add the 3.10 one in this PR?

I mean the 3.11, as the 3.10 is EOL too.

Alpine 3.11 is EOL in Nov 2021 so we'll just have to do it all over again then. It'd be preferable to just jump to the latest, Alpine 3.13 to buy us an extra year.

mthalman · 2021-05-05T12:53:29Z

Looks like the build is failing when executing src/ubuntu/18.04/cross/arm64-alpine/hooks/pre-build

ERROR: unsatisfiable constraints:
  so:libLLVM-11.so (missing):
    required by: lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
                 lldb-11.1.0-r1[so:libLLVM-11.so]
  so:libclang-cpp.so.11.1 (missing):
    required by:
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]
                 lldb-11.1.0-r1[so:libclang-cpp.so.11.1]

mthalman · 2021-05-05T16:37:40Z

The CentOS 7 build is timing out. You'll need to increase the timeout value here:

I suggest a value of 300 for now.

janvorli · 2021-05-06T22:37:02Z

Thanks Matt, I'll fix the timeout and look into the LLVM issue.

janvorli · 2021-07-09T19:47:00Z

@mthalman do you understand the failures in the CI? I am not sure what's the issue. I can see a lot of "keyserver receive failed" messages, but I am not sure if that's the real issue here.

mthalman · 2021-07-09T20:20:45Z

I think the keyserver error message is relevant. Perhaps something has been deprecated in there? I think those commands were lifted from the Node Dockerfiles. Here's the Node 10 Dockerfile: https://github.com/nodejs/docker-node/blob/node-core/10/alpine/Dockerfile. Would be good to compare that to what we have.

The Ubuntu build is failing when running the src/ubuntu/18.04/cross/s390x/hooks/pre-build script. It's attempting to install the libomp5 package for the s390x arch but no such package exists for that arch: https://packages.ubuntu.com/search?suite=bionic&arch=s390x&searchon=names&keywords=libomp. I guess it must have at one point?

janvorli · 2021-07-09T21:30:17Z

@mthalman it seems that node is actually having the same problem with the key servers, see the recent issue nodejs/node#39114. Looks like the keys are gone from the key servers.

janvorli · 2021-07-09T21:38:03Z

A linked issue had a fix, I've made the change in this PR according to that.
nodejs/docker-node#1500

janvorli · 2021-07-13T14:20:25Z

@mthalman the centos-7-graph amd64 leg has failed with the following while building image from src/centos/7/rpmpkg/Dockerfile:

2021-07-12T22:59:37.2657333Z Building native extensions.  This could take a while...
2021-07-12T22:59:52.5655495Z �[91mERROR:  Error installing fpm:
2021-07-12T22:59:52.5656272Z 	git requires Ruby version >= 2.3.

The initial part of the dockerfile installs Ruby 2.0. Any idea what could have caused this issue?
It doesn't seem the centos dockerfiles were touched in a relevant way recently and my change also doesn't do anything with Ruby.

mthalman · 2021-07-13T14:46:21Z

@mthalman the centos-7-graph amd64 leg has failed with the following while building image from src/centos/7/rpmpkg/Dockerfile:
2021-07-12T22:59:37.2657333Z Building native extensions.  This could take a while...
2021-07-12T22:59:52.5655495Z �[91mERROR:  Error installing fpm:
2021-07-12T22:59:52.5656272Z 	git requires Ruby version >= 2.3.
The initial part of the dockerfile installs Ruby 2.0. Any idea what could have caused this issue?

CentOS 7 only includes Ruby 2.0. The latest versions of git are dependent on Ruby >=2.3, which is being pulled in as a dependency of fpm. This is the same kind of issue that we dealt with earlier with ffi (#323). To resolve this, you can pin git to a version that doesn't require Ruby 2.3:

To resolve this, add this

git:1.7.0 \

to

dotnet-buildtools-prereqs-docker/src/centos/7/rpmpkg/Dockerfile

Lines 10 to 12 in 37436dd

    
           && gem install --no-document \ 
        
               ffi:1.12.2 \ 
        
               fpm \

janvorli · 2021-07-13T14:59:19Z

Great, thank you! Let's hope it was the last tweak needed.

We will be switching runtime build to using lld linker

janvorli · 2021-07-13T16:31:21Z

Ah, yet another keyserver issue, this time for Alpine 3.13.

mthalman · 2021-07-13T19:00:19Z

Encountered an error in the Ubuntu build. Here's the last few relevant lines from the log:

2021-07-13T18:48:26.9769680Z cc1plus: warning: -Wabi won't warn about anything [-Wabi]
2021-07-13T18:48:26.9770522Z cc1plus: note: -Wabi warns about differences from the most up-to-date ABI, which is also used by default
2021-07-13T18:48:26.9771351Z cc1plus: note: use e.g. -Wabi=11 to warn about changes from GCC 7
2021-07-13T18:48:26.9772119Z /usr/bin/install: cannot stat 'clearcap.map': No such file or directory
2021-07-13T18:48:26.9772859Z make[2]: [install-clearcap-map] Error 1 (ignored)
2021-07-13T18:48:26.9773687Z --2021-07-13 18:48:19--  https://pkgsrc.joyent.com/packages/SmartOS/2020Q1/x86_64/All/icu-64.2nb2.tgz
2021-07-13T18:48:26.9774330Z Resolving pkgsrc.joyent.com (pkgsrc.joyent.com)... 64.30.129.74
2021-07-13T18:48:26.9774920Z Connecting to pkgsrc.joyent.com (pkgsrc.joyent.com)|64.30.129.74|:443... connected.
2021-07-13T18:48:26.9775498Z HTTP request sent, awaiting response... 301 Moved Permanently
2021-07-13T18:48:26.9776374Z Location: https://us-east.manta.joyent.com/pkgsrc/public/packages/SmartOS/2020Q1/x86_64/All/icu-64.2nb2.tgz [following]
2021-07-13T18:48:26.9777405Z --2021-07-13 18:48:19--  https://us-east.manta.joyent.com/pkgsrc/public/packages/SmartOS/2020Q1/x86_64/All/icu-64.2nb2.tgz
2021-07-13T18:48:26.9778365Z Resolving us-east.manta.joyent.com (us-east.manta.joyent.com)... 64.30.136.21, 64.30.136.19, 64.30.136.18, ...
2021-07-13T18:48:26.9779309Z Connecting to us-east.manta.joyent.com (us-east.manta.joyent.com)|64.30.136.21|:443... connected.
2021-07-13T18:48:26.9779953Z HTTP request sent, awaiting response... 500 Internal Server Error

I'm going to retry the build.

mthalman · 2021-07-13T19:21:58Z

Hmm, the dotnet-buildtools-prereqs-docker-eng build ran into the same error. This occurs when executing the src/ubuntu/18.04/cross/illumos/hooks/pre-build script.

am11 · 2021-07-13T19:39:25Z

I can take a look at illumos failure (pkgsrc url is having some trouble apparently).

am11 · 2021-07-13T20:36:06Z

I have discussed with upstream on IRC (libera.chat #joyent), there is a temporary issue in manta which is being investigated.

Meanwhile, we have a workaround for it to switch to Germany mirror which is intact. @janvorli, could you please add --use-mirror at the end of

dotnet-buildtools-prereqs-docker/src/ubuntu/18.04/cross/illumos/hooks/pre-build

Line 5 in 19bba5c

$SCRIPTPATH/../../../../build-scripts/build-rootfs.sh ubuntu-18.04 illumos x64

I will keep an eye on the server status and switch it back to primary once the issue is resolved.

janvorli · 2021-07-13T20:53:07Z

@am11 I'll add commit with that change once the current legs complete so that I know if there is anything else to fix. The build of the llvm stuff in centos takes a very long time.
Btw, what handles the --use-mirror option? Maybe you know it, but the $SCRIPTPATH/../../../../build-scripts/build-rootfs.sh is not our well known script, but rather a wrapper to that one.

janvorli · 2021-07-13T20:54:43Z

Seems like we'll need to make this script pass through the extra argument to the build-rootfs.sh from arcade.

am11 · 2021-07-13T21:18:09Z

The upstream issue has been resolved. So we don't need this PR to be blocked by that. However, they did mentioned that dotnet is producing a lot of traffic for their server, so we can either parmanently switch to the mirror or switch to OpenIndiana packages (rather than SmartOS ones).

janvorli · 2021-07-14T10:40:30Z

Great, finally the CI is green.

janvorli added the area-dockerfiles label May 5, 2021

janvorli requested a review from mthalman May 5, 2021 00:43

janvorli self-assigned this May 5, 2021

mthalman reviewed May 5, 2021

View reviewed changes

janvorli closed this Jul 12, 2021

janvorli reopened this Jul 12, 2021

janvorli added 5 commits July 13, 2021 09:27

Add lld linker to the centos, alpine and crossbuild images

7a066cb

We will be switching runtime build to using lld linker

Update image build timeouts for the prolonged lld build

e491e24

Fix Alpine Node PGP key servers

9fb1177

Fix fpm git dependency

39117ea

Fix keyservers for Alpine 3.13

9651662

janvorli force-pushed the add-lld-linker branch from d4d1a7f to 9651662 Compare July 13, 2021 16:29

mthalman approved these changes Jul 14, 2021

View reviewed changes

mthalman merged commit 9b5bbc2 into dotnet:main Jul 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lld linker to the centos, alpine and crossbuild images #446

Add lld linker to the centos, alpine and crossbuild images #446

janvorli commented May 5, 2021

mthalman May 5, 2021

janvorli May 5, 2021

janvorli May 5, 2021

mthalman May 5, 2021

mthalman commented May 5, 2021

mthalman commented May 5, 2021

janvorli commented May 6, 2021

janvorli commented Jul 9, 2021

mthalman commented Jul 9, 2021 •

edited

janvorli commented Jul 9, 2021

janvorli commented Jul 9, 2021

janvorli commented Jul 13, 2021

mthalman commented Jul 13, 2021

janvorli commented Jul 13, 2021

janvorli commented Jul 13, 2021

mthalman commented Jul 13, 2021

mthalman commented Jul 13, 2021

am11 commented Jul 13, 2021

am11 commented Jul 13, 2021

janvorli commented Jul 13, 2021

janvorli commented Jul 13, 2021

am11 commented Jul 13, 2021

janvorli commented Jul 14, 2021

Add lld linker to the centos, alpine and crossbuild images #446

Add lld linker to the centos, alpine and crossbuild images #446

Conversation

janvorli commented May 5, 2021

mthalman May 5, 2021

Choose a reason for hiding this comment

janvorli May 5, 2021

Choose a reason for hiding this comment

janvorli May 5, 2021

Choose a reason for hiding this comment

mthalman May 5, 2021

Choose a reason for hiding this comment

mthalman commented May 5, 2021

mthalman commented May 5, 2021

janvorli commented May 6, 2021

janvorli commented Jul 9, 2021

mthalman commented Jul 9, 2021 • edited

janvorli commented Jul 9, 2021

janvorli commented Jul 9, 2021

janvorli commented Jul 13, 2021

mthalman commented Jul 13, 2021

janvorli commented Jul 13, 2021

janvorli commented Jul 13, 2021

mthalman commented Jul 13, 2021

mthalman commented Jul 13, 2021

am11 commented Jul 13, 2021

am11 commented Jul 13, 2021

janvorli commented Jul 13, 2021

janvorli commented Jul 13, 2021

am11 commented Jul 13, 2021

janvorli commented Jul 14, 2021

mthalman commented Jul 9, 2021 •

edited