Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy #162

mz2 · 2023-01-01T21:47:56Z

This might look like a bit of a monster with unrelated changes pooled together. I have no particular attachment to all parts of this ever getting merged, but given the present maturity of the Deb packaging in this repo, there were several interrelated sets of changes that I thought are easiest demonstrated together even if in the end this were cut into smaller separate PRs or only partly accepted. Happy to discuss here, on the forum or wherever 🙂 ☮️

I'll put comments up with further review notes up tomorrow or later this week with better time, but...

The more straightforward changes:

Packages Swift 5.7.2 (i.e. I adjusted the patches for 5.7.2).
Adds jammy build next to focal, sharing resources between them with symlinks (rsync used to turn symlinks into regular files at build time). Changes made here with further symlinking should make supporting additional series simple (control file is really the only resource not shared).

The potentially less straightforward ones:

Adjusts filesystem layout to move away from under /opt/... to /usr/lib/swiftlang/... as one step in preparing to a package layout we could try get sponsored for universe. At least with my interpretation of https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#usrlibexec, this location should be preferable to /usr/libexec, though further changes on this dimension should be straightforward.
Re-divide the package into lib, dev, bin + metapackage, with the separation of the runtime libraries and the compiler executables being the more meaningful change here (the -dev is potentially a bit pointless, or should also contain the lldb related libraries). I do not mean to claim this 3-way split + meta-package precisely should be the division, only meaning to assert that it is more useful than current since the runtime dependencies are comparatively small compared to the rest, and make building Deb or Snap packages that depend on just those possible (which is really where these packages add value compared to just OCI images, IMO).
Associated to above, replace the "plain" symlinks with using the Debian "alternatives", which would make it easy to switch up the /usr/bin/swift etc symlinks between, say, 5.7.2 and some future version, whilst having more than one copy of the toolchain installed (think xcode-select).
Automated deploy into ppa:swiftlang/swiftlang (https://launchpad.net/~swiftlang/+archive/ubuntu/swiftlang), accomplishes right now Jammy and Focal series, for ARM64 and AMD64. I did this initially to make it easy for myself to check that the matrix of series x microarchitectures my changes are affecting continue to build, and in part as another step into producing packages that could eventually be accepted into universe (i.e. separating into a source package build + binary packages getting built on Launchpad). Since I know you state in the upstream README that you intend to firstly create an Apple APT repository, this PPA may not be what you want, but it might be also convenient as that initial step: updates could be signed with your key this way too, it would be easy for people to add the repo (without having to trust a script to modify one's apt sources), and the PPA I created is a "team based" one, which means access could be shared to it for Apple accounts (you could also just take it over, whatever works).

The package can be installed from the PPA as follows on focal or jammy, on either AMD64 or ARM64 (the jammy5 and focal5 versions are the first properly functioning versions).

sudo apt-add-repository ppa:swiftlang/swiftlang
sudo apt install swiftlang # sudo apt install swiftlang-5.7.2-bin libswiftlang-5.7.2 libswiftlang-5.7.2-dev

This is prep for turning the binary packages into libswiftlang, swiftlang-dev, swiftlang (swiftlang-lldb in succession?)

- Re-divide into libswiftlang-X.Y.Z, libswiftlang-X.Y.Z-dev, libswiftlang-X.Y.Z-bin, swiftlang metapackage - Add Jammy build - Share resource files with symlinks between different series where possible

… a self-reference

…e-install and post-removal actions instead.

mz2

Added some hopefully clarifying review notes.

mz2 · 2023-01-02T15:14:54Z

platforms/Linux/DEB/Shared/build_deb.sh

@@ -10,37 +10,66 @@

 set -eux

+source_only_pkg=FALSE # build binary deb by default
+while getopts ":s" option; do


A -s option was added to optionally build a source package with build_deb.sh (source package's assets would then be signed and uploaded to an external build server to build the binary package in case of the PPA or later perhaps an Ubuntu or Debian distribution intended package).

mz2 · 2023-01-02T15:18:29Z

platforms/Linux/DEB/Shared/build_deb.sh


 # build the source package
 ${here}/build_source_package.sh ${staging_dir}

 # install the build dependencies
 cd ${staging_dir}
-mk-build-deps --install ${package_dir}/debian/control.in --tool 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends'
+
+if [ -f /.dockerenv ]; then


This conditional was added since I noticed as part of building in Jammy and Focal LXC containers with non-privileged accounts that it was necessary (and conversely plainly adding sudo would break the Docker based build).

mz2 · 2023-01-02T15:21:27Z

platforms/Linux/DEB/Ubuntu/.dput.cf

@@ -0,0 +1,7 @@
+[swiftlang]


This is a config file needed for uploading the (signed) source package to Launchpad.

mz2 · 2023-01-02T15:24:21Z

platforms/Linux/DEB/Ubuntu/bionic/README.md

@@ -1 +0,0 @@
-## bionic


I took the liberty to remove the bionic build since it didn't exist yet and since bionic is not going to be supported without ESM that much longer (probably not a good practice to encourage building with it since focal and jammy are in existence).

mz2 · 2023-01-02T15:25:59Z

platforms/Linux/DEB/Ubuntu/focal/debian/control.in

+  binutils, git, pkg-config, tzdata, unzip,
+  python3
+Description: Swift programming language
+  Swift is a general-purpose programming language built using


I wasn't very sure if perhaps this longer schpiel about Swift should be repeated in the different packages, or should be there only in the meta-package? If the package division in this style is something you want to keep, the descriptions to the individual packages probably needs revisiting still, either way.

mz2 · 2023-01-02T15:53:43Z

platforms/Linux/DEB/Ubuntu/shared/debian/rules

+SWIFT_BUILDDIR=$(CURDIR)/build
+
+
+export DEB_BUILD_MAINT_OPTIONS=hardening=+format,+fortify,+stackprotectorstrong,+relro,optimize=-lto


These env vars were added as a way to create CFLAGS, CXXFLAGS, LDFLAGS devoid of the -ffat-lto-objects and -flto=auto flags which throws off linking on Jammy otherwise (see https://forums.swift.org/t/how-to-build-the-5-7-1-toolchain-in-swift-org-release-like-archive-layout-on-ubuntu-20-04-or-22-04/61837/8 for my fumbling on this topic).

The inclusion of -flto=auto twice in the strings is intentional; it has to be stripped twice because bizarrely it features twice in the default flags.

Note that although I used these commands to create the list of flags below on lines 83--86 below, the env vars are in fact not referenced below in the override_dh_auto_build (on said lines 83--86) but I instead included hard coded CFLAGS etc down there. I did this because I noticed when dpkg-buildflags --get CFLAGS etc are evaluated in context of debuild, I found that it was producing an empty string for each of these flags (this would successfully build since the offending LTO related flags are then removed, but then leaves out the security oriented flags that are meant to be there for example), whereas when same command is executed in a regular terminal session on the build host, the strings I included in the rules file on lines 83--85 are produced. So I am doing something wrong, and could not quite work out what, yet.

Long story short, it may make sense to just remove these lines 20--22 here unless someone knows how dpkg-buildflags should actually be used to get the wanted effect...

mz2 · 2023-01-02T15:56:20Z

platforms/Linux/DEB/Ubuntu/shared/debian/rules

+	sed -e 's/X.Y.Z/$(SWIFTLANG_PKG_VER)/g' < debian/swiftlang.postinst.in > debian/swiftlang-$(SWIFTLANG_PKG_VER)-bin.postinst
+	sed -e 's/X.Y.Z/$(SWIFTLANG_PKG_VER)/g' < debian/swiftlang.prerm.in > debian/swiftlang-$(SWIFTLANG_PKG_VER)-bin.prerm
+
+	chmod ugo+x debian/swiftlang-$(SWIFTLANG_PKG_VER)-bin.postinst


The postinst and prerm actions were added as a means of installing and removing the alternative symlinks at package install and removal time, respectively. This could alternatively be done with dh_installalternatives also in case of Jammy, but it is too new a feature to exist yet on Focal, I found (that looks to anyway just output similar post-install and pre-removal actions).

mz2 · 2023-01-02T16:08:47Z

platforms/Linux/DEB/Ubuntu/shared/debian/rules

+	# We'd rather not do that either, see above.
+
+override_dh_shlibdeps:
+	dh_shlibdeps -- -xlibswiftlang-$(SWIFTLANG_PKG_VER) -xswiftlang-$(SWIFTLANG_PKG_VER)-bin


The division into libswiftlang-X.Y.Z and libswiftlang-X.Y.Z-bin necessitated:

removing the override_dh_makeshlibs from above (it was overridden to be a no-op previous to these changes).

these -x params passed to dh_slibdeps. In their absence, including ${shlibs:Depends} for these packages in the control file would result in self-references in the Depends lists for these packages (they would list themselves as a dependency). I guess because there are shared libraries that depend on other shared libraries within these packages, and dh_makeshlibs when run mostly fails to list shared library names and versions since SOVERSION is mostly not set anywhere in the Swift libraries built. This is a guess, functionally things seem OK, but I figured further cleanup on this topic would be follow-up material.

Re: making dh_makeshlibs actually produce its output: with the packages divided into a lib and bin package, I did not manage to find flags which would make dh_shlibdeps stage pass without dh_makeshlibs also being run (an alternative would I guess be a shlibs.local file, which gets quite tedious too?)

mz2 · 2023-01-02T16:09:16Z

platforms/Linux/DEB/Ubuntu/shared/debian/rules

@@ -0,0 +1,101 @@
+#!/usr/bin/make -f


This file did not get recognised as a moved file (because original is replaced with a symlink), so I am listing the changes I made systematically below.

mz2 · 2023-01-02T16:17:16Z

platforms/Linux/DEB/Ubuntu/shared/debian/swiftlang.postinst.in

@@ -0,0 +1,7 @@
+#!/bin/sh -e
+
+update-alternatives --install /usr/bin/swift swiftlang /usr/lib/swiftlang/X.Y.Z/bin/swift-frontend 50 \


In the absence of multiple versions of the package that provide an alternative, the automatically installed alternative here gets installed as /usr/bin/swift etc. Once multiple Swift package versions exist, the one exposed as /usr/bin/swift etc is configurable with update-alternatives --config swiftlang.

If this is an acceptable solution, it probably should be documented in the root level README in the repo at least?

mz2 · 2023-01-02T17:01:47Z

... oh, installation instructions for the PPA were also missing, added to description (and below).

sudo apt-add-repository ppa:swiftlang/swiftlang
sudo apt install swiftlang

shahmishal · 2023-01-03T05:31:45Z

Thank you @mz2! I will review these changes soon, and provide my feedback.

shahmishal · 2023-01-17T04:57:26Z

This might look like a bit of a monster with unrelated changes pooled together. I have no particular attachment to all parts of this ever getting merged, but given the present maturity of the Deb packaging in this repo, there were several interrelated sets of changes that I thought are easiest demonstrated together even if in the end this were cut into smaller separate PRs or only partly accepted. Happy to discuss here, on the forum or wherever 🙂 ☮️

@mz2 I think it would good to jump on a call to discuss this pull request, can private message on forums.swift.org?

mz2 · 2023-01-17T13:08:33Z

This might look like a bit of a monster with unrelated changes pooled together. I have no particular attachment to all parts of this ever getting merged, but given the present maturity of the Deb packaging in this repo, there were several interrelated sets of changes that I thought are easiest demonstrated together even if in the end this were cut into smaller separate PRs or only partly accepted. Happy to discuss here, on the forum or wherever slightly_smiling_face peace_symbol

@mz2 I think it would good to jump on a call to discuss this pull request, can private message on forums.swift.org?

DM'd - look forward to speaking with you!

mz2 · 2023-01-19T09:29:11Z

Dumping my notes from the call over here before I forget -- please correct if I have misunderstood anything:

Create a smaller self contained PR to re-target the patches for 5.7.2 (should also contain the hooks that deal with update-alternatives and some README updates that are missing).
Remove GitHub action from the PR to a separate repository that references this one -> not going to be an apple/swift-installer-scripts concern to have this automation, though you may find it a useful reference to build a similar automation at your end. I think I will associated with that change create in that other repo that I'll maintain a Launchpad build recipe to create the PPA that I have demonstrated here. PPA itself becomes a temporary resource, useful until you have your apt repository up (... until we have a universe grade package ready hopefully later 🤞 ).
Create a separate branch that accomplishes the libswiftlang / swiftlang split, @shahmishal to consider the package boundaries (is a separate -dev useful, perhaps for sourcekit-lsp or other toolchain executable independent use?), perhaps also to drive discussion of this on forum?
Revert the Platform: any changes back to "amd64 arm64".
The docker-compose yamls probably can be made one with the help of arguments, similar to what's done with the RPMs.

5hv5hvnk · 2023-02-13T06:15:15Z

Hi, can I get try continue this?

mz2 · 2023-02-16T13:18:09Z

@5hv5hvnk I'm about to post changes either tomorrow or later this weekend on the lines noted above. I'd propose waiting until that 😄 Sorry for the delay.

MaxDesiatov · 2023-03-31T11:55:35Z

Hi @mz2, do you intend to proceed with this PR? Maybe you could provide some details for what's missing in this PR before it can be reviewed?

mz2 · 2023-04-05T20:22:13Z

Yep, indeed am going to, just spring flu and work stuff of managerial kind got in the way of hacking time in the end recently. I’ll have another stab at splitting this on the easter weekend.

On Fri, 31 Mar 2023 at 14:55 Max Desiatov ***@***.***> wrote: Hi @mz2 <https://github.com/mz2>, do you intend to proceed with this PR? Maybe you could provide some details for what's missing in this PR before it can be reviewed? — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARNQZW6ZWPNQI6KWPPEBLW63A4FANCNFSM6AAAAAATOKTAVM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- - - - - - Matias Piipari, PhD ***@***.***

mz2 added 4 commits January 1, 2023 23:14

Fix up the patches for 5.7.2

aaa7206

Drops the -X.Y.Z package as prep for re-dividing into lib, dev, bin

4eebf68

This is prep for turning the binary packages into libswiftlang, swiftlang-dev, swiftlang (swiftlang-lldb in succession?)

Build and upload a signed source package to Launchpad using a GH action

8d0350e

Divide into lib, dev, bin + metapackage and add Jammy build

8b38ef9

- Re-divide into libswiftlang-X.Y.Z, libswiftlang-X.Y.Z-dev, libswiftlang-X.Y.Z-bin, swiftlang metapackage - Add Jammy build - Share resource files with symlinks between different series where possible

mz2 changed the title ~~Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib, dev, bin + metapackage, automated ppa:swiftlang/swiftlang deploy~~ Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy Jan 1, 2023

mz2 marked this pull request as ready for review January 1, 2023 21:48

mz2 requested a review from shahmishal as a code owner January 1, 2023 21:48

mz2 mentioned this pull request Jan 1, 2023

Adjust Ubuntu related patches to build 5.7.1 #149

Draft

mz2 added 9 commits January 2, 2023 14:17

Fix to installing files to swiftlang-X.Y.X-bin

e300a23

Exclude swiftlang-X.Y.Z-bin from shared library dependencies to avoid…

7dd1572

… a self-reference

Change from a branch to a tag based trigger

febb450

Remove some redundant env vars from the GH workflow

c367a8c

Add back --no-install-recommends

579af96

Remove --no-lintian from the source package build case

a30222a

Remove optimize=-lto which lintian warns about (I think on focal only?)

d4a372f

Fixes to some debug logging in debian/rules

ccfce2e

Remove swiftlang.alternatives.in since alternatives are managed by pr…

9e25fbd

…e-install and post-removal actions instead.

mz2 commented Jan 2, 2023

View reviewed changes

tomerd mentioned this pull request Jan 18, 2023

Add control files for Debian packages #161

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy #162

Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy #162

mz2 commented Jan 1, 2023 •

edited

mz2 left a comment

mz2 Jan 2, 2023

mz2 Jan 2, 2023

mz2 Jan 2, 2023

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023

mz2 commented Jan 2, 2023

shahmishal commented Jan 3, 2023

shahmishal commented Jan 17, 2023

mz2 commented Jan 17, 2023

mz2 commented Jan 19, 2023

5hv5hvnk commented Feb 13, 2023 •

edited

mz2 commented Feb 16, 2023

MaxDesiatov commented Mar 31, 2023

mz2 commented Apr 5, 2023 via email

		SWIFT_BUILDDIR=$(CURDIR)/build


		export DEB_BUILD_MAINT_OPTIONS=hardening=+format,+fortify,+stackprotectorstrong,+relro,optimize=-lto

		@@ -0,0 +1,7 @@
		#!/bin/sh -e

		update-alternatives --install /usr/bin/swift swiftlang /usr/lib/swiftlang/X.Y.Z/bin/swift-frontend 50 \

Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy #162

Are you sure you want to change the base?

Package 5.7.2 for Ubuntu (focal, jammy), re-divide into lib / dev / bin + metapkg, automated ppa:swiftlang/swiftlang deploy #162

Conversation

mz2 commented Jan 1, 2023 • edited

mz2 left a comment

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 Jan 2, 2023 • edited

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 Jan 2, 2023 • edited

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 Jan 2, 2023 • edited

Choose a reason for hiding this comment

mz2 Jan 2, 2023 • edited

Choose a reason for hiding this comment

mz2 Jan 2, 2023

Choose a reason for hiding this comment

mz2 commented Jan 2, 2023

shahmishal commented Jan 3, 2023

shahmishal commented Jan 17, 2023

mz2 commented Jan 17, 2023

mz2 commented Jan 19, 2023

5hv5hvnk commented Feb 13, 2023 • edited

mz2 commented Feb 16, 2023

MaxDesiatov commented Mar 31, 2023

mz2 commented Apr 5, 2023 via email

mz2 commented Jan 1, 2023 •

edited

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023 •

edited

mz2 Jan 2, 2023 •

edited

5hv5hvnk commented Feb 13, 2023 •

edited