Allow extensions to be compiled from GitHub sources

[alcaeus]

---

name: 🎉 New Feature
about: Allow extensions to be compiled from GitHub sources
labels: enhancement

---

A Pull Request should be associated with a Discussion.

Note: no discussion available as I built this for my own needs and want to start a discussion based on the changes I made.

Description

This PR allows installing extensions from source. This is done by using a special extension format of the form <extension>-<org>/<repo>@<version>. Currently, only GitHub is supported and the repository is checked out using git clone --recurse-submodules -b <version> https://github.com/<org>/<repo>. This is supported on Linux and OSX, but not on Windows environments.

The rationale for adding this was that I didn't want to have separate steps depending on what extension version to install. This way, I can provide the repository URL and branch name as version string, allowing installing stable versions and compiling from source in a single step in a matrix. This was inspired by the leftover install_extension_from_source, which this PR also removes as it's unused (and replaced by this PR).

• I have written test cases for the changes in this pull request
• I have run npm run format before the commit.
• I have run npm run lint before the commit.
• I have run npm run release before the commit.
• npm test returns with no unit test errors and all code covered.
• I have checked the edited scripts for syntax.
• I have tested the changes in an integration test (linux only, can run on OSX separately): https://github.com/alcaeus/mongo-php-library/runs/1741774149 (also, please ignore the PHPUnit failure - that is unrelated).

Further discussion

• Should we allow passing commit hashes as well? This is not supported due to using git clone but could be supported as well.
• Should we provide way for users to customise the build, e.g. arguments to pass to configure? If so, this might require more changes to the action (including potentially changing the format of the extensions input)
• I have yet to figure out whether this will play nice with shivammathur/cache-extensions. I assume there will be issues if somebody installed ext-org/repo@master and the commit hash changes before the cache is invalidated. This needs documentation at the very least. This can be worked around by supporting commit hashes (see above).

Let me know if you have any concerns about adding this, I'm happy to talk about potential alternatives.

[shivammathur]

@alcaeus Thanks for working on this. Installing extensions from source was on the roadmap. 🚀

• Yes, it would be a good idea to support commit hashes as well.
• I do not think it would be a good API if in extensions input, users are allowed to pass configuration arguments. An alternative would be checking for an env variable say MONGODB_CONFIGURE_ARGS for mongodb extension.
• Another improvement can be checking for required libraries in env variable MONGODB_LIBS.
• As there is no check in add_extension_from_github to see if extension is already in extension_dir, extensions cached by extensions-cache will be overwritten by new compiled one.

[jmikola]

An alternative would be checking for an env variable say MONGODB_CONFIGURE_ARGS for mongodb extension.

@shivammathur that sounds like a great idea, as it presumably makes it much easier to customize configure args via the build matrix. Granted, we're more likely to want to do that for the extension's own builds than a consuming library, but still nice to have the option.

[alcaeus]

@shivammathur thanks for your feedback. I've applied the suggestions from code review. Please let me know if the checks in extensions.ts can be simplified - I'm not entirely happy with the duplicate regex evaluation, especially since that code can't be covered.

• I've changed add_extension_from_github to accept a commit hash as well. Anything accepted by git checkout is now fair game as a version.
• I've played around with supporting <ext_name>_CONFIGURE_ARGS - this isn't a lot of effort but the question remains whether we want a common solution for all extensions or whether extensions should take care of this themselves.
• An optional LIBS env var is possible, but I wonder if it would be more sensible for users to fill build dependencies themselves
• Skipping the cache is fine for now. I checked how this is handled for pecl, and since the only "version" we can reliably use is a commit hash which is lost after installing, extensions are always rebuilt from source, even if the cache restored them. I can revisit this separately if we want the cache to pick up source builds as well.

[shivammathur]

@alcaeus Awesome,

• You can remove the if(matches == null) blocks in extension.ts as those are unreachable. That should fix the coverage.
• Support for args and libs is not required and can be added in future. I'm happy to merge this PR without that.
• Support for cache can be added, since the cache key depends on the extensions input, it would invalidate when commit sha or version changes in the workflow. (https://github.com/shivammathur/cache-extensions/blob/master/src/extensions.sh#L95)

[alcaeus]

@shivammathur Just a heads up, it will be a few days before I can return to this. I haven't forgotten but have to rest for a few days.

[shivammathur]

@alcaeus Take your time. Hope you are feeling well.

[alcaeus]

@shivammathur thanks! I've rebased on develop and removed the superfluous if statements. I'll take a look at cache next, but believe that the PR can be merged as-is.