HTTP selfupdate #184

neverpanic · 2020-06-06T15:59:25Z

Note: A few things still need to happen before this can be merged:

Generate and upload signatures for the current release to GitHub and distfiles
Possibly adjust CI to generate the new signature format
Adjust the releasing guide to cover the new steps required
Add a changelog entry
Test signify compilation on older macOS (help wanted!)
Commit _resources/port1.0/fetch/base_mirror_sites.list

I'm uploading this so you can have a look and see if you notice any issues with my approach. Note that this should be reviewed commit-by-commit – otherwise you'll get a huge patchset that's not simple to review.

Description

Refactor the selfupdate package into smaller functions and switch to a control flow of:

checking for newer versions by downloading a single URL
downloading the new version using HTTP when available
verifying the signature using EdDSA with ed25519 elliptic curve crypto with the help of OpenBSD's signify(1)
automatically re-executing selfupdate for the sync step with the new base version when base was updated

This allows us to automatically fall back to other mirrors should our main mirror not be available. We did not have this functionality available for our previous rsync-based mechanism.

Additionally, the use of our standard curl wrapper enables displaying a progress bar for the download and automatically uses any proxies that might already be configured for HTTP.

Since we currently do not have a standard mechanism to sign the source code tarballs uploaded to our distfiles server and github releases page (we only sign the tarball pushed to rsync), I've taken the liberty to introduce a new modern signature scheme for this channel – if we have to touch this anyway, we might as well use modern tools and algorithms.

Previously, we would instruct users to re-run selfupdate manually when MacPorts base was updated, because Portfiles might already require the new base version to index correctly. This step has now been automated using an idea that lives on the migration branch, where the port.tcl client automatically re-executes itself in this situation.

This patch also adds support for a list of mirrors specified in the ports tree at _resources/port1.0/fetch/base_mirror_sites.list, so that we can add and remove mirrors without releasing a new version of MacPorts base. Base itself comes preconfigured with the GitHub releases URL if this list is not available.

See: https://trac.macports.org/ticket/60608

@variable

We did define those, but in the places where they could have been used, the @variable@ syntax was used instead, so we can just drop those.

signify(1) is a tool originally developed for OpenBSD that uses modern EdDSA signatures on the ed25519 elliptic curve. Bundling this tool gives us a good way forward away from our single RSA 4096 public key that has now been in use since 2011 for all packages, all base updates and all ports trees. Shipping signify allows us to use (a) modern cryptography, (b) avoid using the same key for everything and (c) eventually phase out the RSA-RIPEMD160 signatures. The vendor/signify-osx directory is a clone of https://github.com/jpouellet/signify-osx/, udpated with the latest OpenBSD upstream signify source code from CVS using the 'make up' target of the signify-osx Makefile.

Add a new public key for use with signify(1) to verify downloaded MacPorts base relases. Adjust the Makefile to install this key in $prefix/share/macports/keys/base, so that we have a designated space for additional keys (enabling future key roll-over) and separate keys, for example for a ports tree signature.

We should not hardcode the location of the keys directory or the signify binary in any source code files, so add a autoconf variable that can be used instead.

This list of options was hard to read, hard to maintain, hard to keep sorted, and hard to diff in its current form. Move every option name to a line of its own and sort to simplify this in the future.

Currently, all of our configuration file options hold a single value only, and while we could space-separate them, this quickly gets messy due to the lack of line continuation support. Introduce a new type of configuration option than can be specified multiple times in one configuration file and appends the new value to a list, instead of overwriting the old value. Note that this append functionality happens per configuration file, to enable overwriting the defaults specified in one configuration file in the next one, rather than only appending.

To support updating MacPorts base via HTTP and avoiding the rsync download (which we currently do not distribute over multiple mirrors automatically), introduce two new multi-value configuration options release_version_url and release_url, where the former is a list of URLs to check for the current MacPorts release version, and the latter is a list of template URLs where to download the source code for a given MacPorts version. See: https://trac.macports.org/ticket/60608

Refactor the selfupdate package into smaller functions and switch to a control flow of: 1. checking for newer versions by downloading a single URL 2. downloading the new version using HTTP when available 3. verifying the signature using EdDSA with ed25519 elliptic curve crypto with the help of OpenBSD's signify(1) 4. automatically re-executing selfupdate for the sync step with the new base version when base was updated This allows us to automatically fall back to other mirrors should our main mirror not be available. We did not have this functionality available for our previous rsync-based mechanism. Additionally, the use of our standard curl wrapper enables displaying a progress bar for the download and automatically uses any proxies that might already be configured for HTTP. Since we currently do not have a standard mechanism to sign the source code tarballs uploaded to our distfiles server and github releases page (we only sign the tarball pushed to rsync), I've taken the liberty to introduce a new modern signature scheme for this channel – if we have to touch this anyway, we might as well use modern tools and algorithms. Previously, we would instruct users to re-run selfupdate manually when MacPorts base was updated, because Portfiles might already require the new base version to index correctly. This step has now been automated using an idea that lives on the migration branch, where the port.tcl client automatically re-executes itself in this situation. This patch also adds support for a list of mirrors specified in the ports tree at _resources/port1.0/fetch/base_mirror_sites.list, so that we can add and remove mirrors without releasing a new version of MacPorts base. Base itself comes preconfigured with the GitHub releases URL if this list is not available. See: https://trac.macports.org/ticket/60608

Those options were used for rsync-based selfupdate, but this is no longer required since MacPorts now uses HTTP to update itself. See: https://trac.macports.org/ticket/60608

jmroot

This all looks pretty reasonable for the most part. Two issues are:

The scope of this PR is quite large, covering multiple features: switching selfupdate from rsync to HTTPS, adding signify and using it for signature verification, and re-executing after updating base to continue with syncing the ports tree. Some of it can't be merged yet. It might be better to split it up so some can be merged sooner?
The new selfupdate method is going to be very difficult to adequately test without doing an actual release. I think it would be safer to start by adding the HTTPS method but use rsync by default for the first release, so developers and interested users can try out HTTPS selfupdate under real world conditions, but everyone else doesn't break if there are bugs. Then for the next release, make HTTPS the default, then if all goes well, remove rsync in the next release.

jmroot · 2020-10-19T22:14:21Z

src/macports1.0/macports.tcl

                        set macports::$option [string trim $val]
                        global macports::$option
+                    } elseif {$option in $bootstrap_multivalue_options} {


I'm not terribly happy about having two different behaviours for options in the config file. Is it just to avoid long lines?

Yes. Do you have a better suggestion on how to achieve this?

jmroot · 2020-10-19T22:19:24Z

vendor/signify-osx/src/CVS/Entries

@@ -0,0 +1 @@
+D


Might want to add the CVS stuff to .gitignore.

Good idea, yes.

neverpanic

We could merge everything up to and including "macports1.0: Sort and format list of options", and merge the rest later?

I like the idea of having this available selectively first, but I fear it would not really give us good test coverage unless we make it the default and fall back to rsync if it fails. I can take a look at implementing that, if you want?

neverpanic · 2020-10-20T11:07:36Z

src/macports1.0/macports.tcl

                        set macports::$option [string trim $val]
                        global macports::$option
+                    } elseif {$option in $bootstrap_multivalue_options} {


Yes. Do you have a better suggestion on how to achieve this?

neverpanic · 2020-10-20T11:10:08Z

vendor/signify-osx/src/CVS/Entries

@@ -0,0 +1 @@
+D


Good idea, yes.

jmroot · 2020-10-22T11:36:52Z

We could merge everything up to and including "macports1.0: Sort and format list of options", and merge the rest later?

Sounds good. Remember to add the .gitignore changes first.

I like the idea of having this available selectively first, but I fear it would not really give us good test coverage unless we make it the default and fall back to rsync if it fails. I can take a look at implementing that, if you want?

That would be even better I guess. Probably still have a command-line option to make it use rsync, just in case.

herbygillot · 2022-02-15T17:40:50Z

Can we revisit this?

Would it be possible to test base + bundled signify using the buildbots?

neverpanic · 2022-02-15T20:30:15Z

This needs spitting into smaller parts, and some refactoring to keep the rsync selfupdate as fallback. I haven't found the time to implement that yet, unfortunately.

vendor: Drop unused variables from Makefile.in

d54a6e8

We did define those, but in the places where they could have been used, the @variable@ syntax was used instead, so we can just drop those.

neverpanic added enhancement help wanted labels Jun 6, 2020

neverpanic requested review from raimue, ryandesign and jmroot June 6, 2020 15:59

neverpanic force-pushed the cal-http-selfupdate branch from fb37b9a to 6b1bb2c Compare June 6, 2020 16:13

neverpanic added 10 commits June 6, 2020 18:16

macports1.0: Sort variables in macports::autoconf

a5b2cb8

macports1.0: Add signify, key directory path vars

13abc3c

We should not hardcode the location of the keys directory or the signify binary in any source code files, so add a autoconf variable that can be used instead.

macports1.0: Sort and format list of options

a3e1cb0

This list of options was hard to read, hard to maintain, hard to keep sorted, and hard to diff in its current form. Move every option name to a line of its own and sort to simplify this in the future.

macports.conf: Drop rsync_server, rsync_dir

53bac7b

Those options were used for rsync-based selfupdate, but this is no longer required since MacPorts now uses HTTP to update itself. See: https://trac.macports.org/ticket/60608

doc: Re-generate manpages

4fdcf6f

neverpanic force-pushed the cal-http-selfupdate branch from 6b1bb2c to 4fdcf6f Compare June 6, 2020 16:16

jmroot reviewed Oct 19, 2020

View reviewed changes

neverpanic commented Oct 20, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP selfupdate #184

HTTP selfupdate #184

neverpanic commented Jun 6, 2020 •

edited by herbygillot

jmroot left a comment

jmroot Oct 19, 2020

neverpanic Oct 20, 2020

jmroot Oct 19, 2020

neverpanic Oct 20, 2020

neverpanic left a comment

neverpanic Oct 20, 2020

neverpanic Oct 20, 2020

jmroot commented Oct 22, 2020

herbygillot commented Feb 15, 2022

neverpanic commented Feb 15, 2022

HTTP selfupdate #184

Are you sure you want to change the base?

HTTP selfupdate #184

Conversation

neverpanic commented Jun 6, 2020 • edited by herbygillot

Description

jmroot left a comment

Choose a reason for hiding this comment

jmroot Oct 19, 2020

Choose a reason for hiding this comment

neverpanic Oct 20, 2020

Choose a reason for hiding this comment

jmroot Oct 19, 2020

Choose a reason for hiding this comment

neverpanic Oct 20, 2020

Choose a reason for hiding this comment

neverpanic left a comment

Choose a reason for hiding this comment

neverpanic Oct 20, 2020

Choose a reason for hiding this comment

neverpanic Oct 20, 2020

Choose a reason for hiding this comment

jmroot commented Oct 22, 2020

herbygillot commented Feb 15, 2022

neverpanic commented Feb 15, 2022

neverpanic commented Jun 6, 2020 •

edited by herbygillot