Option to vendor dependencies to easy bootstrapping

[rossburton]

Would it be possible to have a version of build which vendors its dependencies?

The bootstrap cycle isn't fun and I'm trying to reduce the number of recipes that we have to build by hand. We've a generic 'pep 517' building class that uses build and installer but obviously that leads to dependency cycles involving build, installer, pep517, tomli.

I've actually bitten the bullet and written a minimal clone of build that vendors pep517 and tomli as a prototype, and considering our scope is smaller (no isolated builds are needed) it's almost fully functional at 58 lines.

However, I'd prefer not to maintain code. A build which embedded it's dependencies would be easier for us, and all other source-based distributions.

[gaborbernat]

This has been discussed before and we decided not to go down this path. We already decided to switch to flit that's a dependency free build backend.

[rossburton]

Is there a timeline for switching to flit?

[henryiii]

We need at least one release (maybe today, very soon), then packaging needs to move to flit (which I'm helping with now), and not sure about pyparsing, since packaging depends on that. Then we'll implement it here.

[layday]

Closing in favour of #394, PR at #471 when the time's ripe.

[adamjstewart]

Apologies for reviving a dead discussion, but I would also like to second this.

I'm a developer for the Spack package manager. At the moment, Spack uses pip to install all of its Python packages, but we would really like to switch to build/installer. Since pip vendors all of its dependencies, bootstrapping pip only requires me to install the following packages manually:

• pip
• wheel

However, bootstrapping build requires me to install (depending on OS and Python version):

• build
• wheel?
• packaging
  • pyparsing
• pep517
  • zipp
• colorama
• importlib-metadata
  • typing-extensions
• tomli

If these dependencies could be vendored, it would make bootstrapping significantly easier.

@pradyunsg may be interested in this discussion.

[henryiii]

Wheel (and setuptools, which you didn't mention) will be going away (albeit being replaced by flit-core, which has no dependencies, it only vendors tomli). Packaging dropped pyparsing and will use flit-core too. Also importlib-metadata, typing-extensions, and even tomli will go away on their own. Colorama is optional. So that leaves for Python 3.11:

• flit-core
• packaging
• pep517 -> pyproject-hooks
  • zipp

(+ a few things for older Pythons)

Also, just curious, but is there any reason other than historical that you can't use installer + a wheel to get build instead of an SDist? (It's still source, it's pure Python)

[layday]

pyproject-hooks has dropped the dependency on zipp. Sent with [Proton Mail](https://proton.me/) secure email.

…

------- Original Message -------

On Sunday, November 27th, 2022 at 06:25, Henry Schreiner ***@***.***> wrote: Wheel (and setuptools, which you didn't mention) will be going away (albeit being replaced by flit-core, which has no dependencies, it only vendors tomli). Packaging dropped pyparsing and will use flit-core too. Also importlib-metadata, typing-extensions, and even tomli will go away on their own. Colorama is optional. So that leaves for Python 3.11: - flit-core - packaging - pep517 -> pyproject-hooks - zipp (+ a few things for older Pythons) Also, just curious, but is there any reason other than historical that you can't use installer + a wheel to get build instead of an SDist? (It's still source, it's pure Python) — Reply to this email directly, [view it on GitHub](#470 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWCGB7FFB5SQ6MWIFS3WKLPE5ANCNFSM5VZGOB6Q). You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[rossburton]

@adamjstewart Last year I decided to write an interim solution for our use which is a very minimal build clone with no external dependencies until the dependency tree is smaller.

https://gitlab.com/rossburton/picobuild

it may be missing functionality that you need: for example it doesn’t fetch dependencies as it is designed for building a traditional Linux distribution. But it might be useful?

note that it will be abandoned once the dependency tree is sufficiently small and build can be used directly without too much effort.

[adamjstewart]

is there any reason other than historical that you can't use installer + a wheel to get build instead of an SDist?

For context, Spack is a from-source package manager. We try to build everything directly from source when possible, although we're okay with building things from binaries like wheels when needed to prevent circular dependencies. We use wheels to install pip and wheel for this reason. The packages I listed above are all of the packages we would need to install from wheel just to get build working. My point is not that it isn't possible to bootstrap build, but that it's significantly more cumbersome than bootstrapping pip.

Also importlib-metadata, typing-extensions, and even tomli will go away on their own. Colorama is optional... (+ a few things for older Pythons)

I don't think it's that easy to sweep these under the rug. Spack finally dropped Python 2 support last week after a multi-year battle to convince everyone to abide by Python's EOL cycle. We certainly aren't going to drop 3.7 any time soon, and 3.11+ won't be the case until 2026 at the earliest.

it may be missing functionality that you need: for example it doesn’t fetch dependencies as it is designed for building a traditional Linux distribution.

This is actually exactly what we're looking for. We need to be able to build on air-gapped networks, so we would actually prefer the ability to prevent build from downloading and installing dependencies. I'm a bit hesitant to use this fork since it may quickly start to diverge from upstream, but I would be very happy if build itself did the same thing.

[rossburton]

Picobuild isn’t a fork, and I’ll be maintaining it for our use case until it can be replaced by build. That won’t be happening in the short term :)

[rossburton]

(Feel free to email me if you want to discuss picobuild, ross@burtonini.com)

[layday]

You can run build with just pep517 (soon to be pyproject-hooks) if that's what you want, or you can maintain an internal build "lite" package without any of the other dependencies for the purpose of bootstrapping. Although we currently don't, I'd personally be ok with us guaranteeing that build can be used to build packages without isolation without requiring any dependencies other than pyproject-hooks. If we need to provide an alternative build spec (pyproject.toml) to make that easier, it's something that I would at least be willing to consider.

[adamjstewart]

You can run build with just pep517 (soon to be pyproject-hooks) if that's what you want

Are you suggesting that all other dependencies for all OSes and all versions of Python are optional and build will still work without them?

[layday]

For Python < 3.11, you'll also need tomli, but yes, the remaining dependencies are optional.

[rossburton]

@layday i may get back to you on that once pyproject-hooks has released :)

[rossburton]

You can run build with just pep517 (soon to be pyproject-hooks) if that's what you want, or you can maintain an internal build "lite" package without any of the other dependencies for the purpose of bootstrapping. Although we currently don't, I'd personally be ok with us guaranteeing that build can be used to build packages without isolation without requiring any dependencies other than pyproject-hooks. If we need to provide an alternative build spec (pyproject.toml) to make that easier, it's something that I would at least be willing to consider.

I don't think this will be possible, check_dependency() uses packaging.requirements: https://github.com/pypa/build/blob/main/src/build/__init__.py#L148

[layday]

You can disable the dependency check with -x.

[adamjstewart]

Is it possible to enable dependency check but disable automatic download and installation of dependencies? Or do those come hand-in-hand like with pip install --no-deps?

[layday]

With isolation dependencies are downloaded but not checked. Without isolation (-n) dependencies are not downloaded but they are checked by default. Pass -nx to neither download dependencies nor check them.

[adamjstewart]

@pradyunsg is that also how it works with pip? Should we be using --no-build-isolation instead of --no-deps --no-build-isoloation in order to skip dependency download but still ensure that they are already installed? I also remember having issues where pip wouldn't install the package if it was already found in PYTHONPATH but I think --ignore-installed solved that one.

[adamjstewart]

FWIW, I'm trying to bootstrap build without pip in spack/spack#35326 and found that it requires installing no less than 15 packages from wheels in order to avoid circular dependencies. I know there isn't much interest here in vendoring dependencies, but I would once again like to request that build vendor its dependencies like pip, flit, and most other build tools do. Otherwise, it's prohibitively difficult to install build.

[pradyunsg]

You need to pass --check-build-dependencies to pip for it to check the build-time dependencies. This check is redundant if you haven't disabled isolation.

[pradyunsg]

I think you only need packaging and pyproject-hooks for running build; it operates cleanly without the rest.

[henryiii]

• installer (build-time only)
• flit-core (build-time only)
  • toml (3.1–3.3 only)
  • pytoml (–3.0 only)
• packaging
  • pyparsing (–21 only)
  • six (–20.7 only)
  • attrs (19.1 only)
• colorama (Windows only)
• importlib-metadata (Python 3.7 only)
  • zipp
  • typing-extensions (Python 3.7 only)
• tomli (Python 3.7–3.10 only)

I don't understand your dependency list. pyproject-hooks is missing. flit-core vendors tomli, so why do you need toml/pytoml? Colorama is optional, just ignore it. Packaging dropped pyparsing and six and attrs - why do you care about older versions of packaging and flit-core? If that's the case, what's the point of a new release of build with vendoring, wouldn't you want to support older build versions too?

So the actual dependency list is:

• installer (build-time only)
• flit-core (build-time only)
• packaging
• importlib-metadata (Python 3.7 only)
  • zipp
  • typing-extensions (Python 3.7 only)
• tomli (Python 3.7–3.10 only)
• pyproject-hooks

That's 8 things, or 5 if you drop Python 3.7, which happens later this year. Only 4 things for the latest Python. Two of them are build-time only.

[adamjstewart]

pyproject-hooks is missing

Fixed.

flit-core vendors tomli, so why do you need toml/pytoml?

flit-core 3.6+ does, older versions don't.

why do you care about older versions of packaging and flit-core?

ctgan, galaxy-util, poetry, and torchgeo all require older versions of packaging.

colorio, pkgutil-resolve-name, and testpath all require older versions of flit-core.

If that's the case, what's the point of a new release of build with vendoring, wouldn't you want to support older build versions too?

We don't currently have any packages that require older versions of build, so we could theoretically remove older versions quite easily.

or 5 if you drop Python 3.7, which happens later this year.

We're planning on deprecating 3.7 in June but realistically won't be able to remove it entirely until December.

If it weren't for the patch from #406 that we require, we could drop flit-core too and install from a wheel of build. Hoping this bug is fixed someday...

I think you only need packaging and pyproject-hooks for running build; it operates cleanly without the rest.

Let me try this. I'm a bit wary of excluding dependencies listed as required (I think pip check would fail for example) but it may be more palatable than my current PR.

[henryiii]

But do you need to install the old version of packaging & flit-core to build build? You can build build with the newer packaging then after that, you are free to use it however you want, including to build older versions of packaging and flit-core. I'd also push those packages to see if they can update to use the latest packaging/flit-core, as requiring older versions of those is very bad, especially for newer or more exotic systems, and should be avoided.

[layday]

https://flit.pypa.io/en/stable/bootstrap.html describes the bootstrapping process with flit operating as both the frontend and the backend.

[henryiii]

Poetry is unclamped. They clamp a lot, but not on packaging. ctgan is capping everything, I'd recommend opening an issue with them and linking to my post and this issue. Same probably for all the others, I just checked the first.

[adamjstewart]

But do you need to install the old version of packaging & flit-core to build build? You can build build with the newer packaging then after that, you are free to use it however you want, including to build older versions of packaging and flit-core.

At the moment Spack doesn't support a package appearing multiple times in the DAG, so you can't have a situation like:

foo -> packaging (old) -> build -> packaging (new)

@alalazo is working on better support of this, but it's not ready yet.

I'd also push those packages to see if they can update to use the latest packaging/flit-core, as requiring older versions of those is very bad, especially for newer or more exotic systems, and should be avoided.

Completely agree, but something we have very little control over. I'm happy to chase people down and try to get them to support newer versions of dependencies, but from issue to PR to new version release could be a 6+ month cycle depending on how frequent software releases are. Many of the packages in Spack haven't been updated in decades (we just dropped a ton of packages when we dropped Python 2 support).

Poetry is unclamped. They clamp a lot, but not on packaging.

Poetry 1.2+ is unclamped on packaging, poetry 1.1 and older is clamped. We don't currently have any packages that explicitly require poetry 1.1 and older, so we could remove this though.

Spack values reproducible builds, so we try not to remove old versions of packages that a user may rely on. I personally care far more about supporting the latest and greatest and keeping packages simple, so it's been a tug-of-war to push forward with things like this. If it was up to me, Spack would only support Python 3.11+ and the latest version of every package. But it's not up to me, and probably for good reason 😄

[adamjstewart]

@henryiii I actually disagree with https://iscinumpy.dev/post/bound-version-constraints/ but don't want to derail this conversation with an aside.

[adamjstewart]

I think you only need packaging and pyproject-hooks for running build; it operates cleanly without the rest.

importlib-metadata seems to be required for Python 3.7 if you want to check dependencies (we do). I'm guessing tomli is required for Python 3.10 and older for any project that uses a pyproject.toml file. I don't have Windows so I can't confirm or deny that colorama is optional but it's only 1 package so that's not too bad. But yeah, still seems like we have 8+ packages in this circular dependency loop. Would love to get that down to 0 someday.

[FFY00]

The only dependency required to bootstrap build is pyproject-hooks and flit-core. See https://pypa-build.readthedocs.io/en/stable/installation.html#bootstrapping (it is outdated, toml is not needed anymore because tomllib is now in the standard library, and pep517 has been renamed to pyproject-hooks).

You can do:

• flit-core (run build_dists.py or python -c "from flit_core import buildapi; buildapi.build_wheel('dist/')", and then bootstrap_install.py to install the built wheel)
• installer (run python -c "from flit_core import buildapi; buildapi.build_wheel('dist/')", and then python -m installer to install the built wheel)
• pyproject-hooks (run python -c "from flit_core import buildapi; buildapi.build_wheel('dist/')", and then python -m installer)
• (if you're on Python < 3.11, repeat the pyproject-hooks step for tomli)
• build (run python -m build -nx, and then python -m installer)

After that, you can use python -m build -nx and python -m installer to build everything else. After you have built all dependencies of build, you can drop the -x.

Just then beware that build backends might have dependency cycles, so you'll need to sort that out with the according upstreams.

importlib-metadata seems to be required for Python 3.7 if you want to check dependencies (we do).

Build an initial version of build without that dependency, then build all missing dependencies using -x, and finally build a new version of build with that specifies all dependencies in the metadata.

I should probably improve the documentation regarding this.

[adamjstewart]

The only dependency required to bootstrap build is pyproject-hooks and flit-core.

I would like to once again remind everyone that not only do Python 3.7–3.10 exist, but that we need to continue to support them until at least 2026 when Python 3.10 reaches EOL. Pretending that dependencies don't exist because they aren't required on Python 3.11 or Linux isn't an option for us.

[bootstrapping build is simple and easy]

[proceeds to describe a 10+ step process that involves building build 3 separate times with an increasing list of dependencies required for normal functionality]

I really like the design philosophy behind build and installer. It follows the Unix philosophy of "do one thing well". I use build when generating release files for all of the libraries I maintain. And I would love to increase the adoption of build and installer in package managers like Spack.

But the process you've described above is prohibitively difficult, and I can't imagine that even the developers of build use this process to install build. I'm starting to come to the conclusion that the only realistic way to install build is with pip, and that while it is technically possible to bootstrap without pip, this process is neither recommended nor supported. And if pip is required to install build, then there's no reason to use it in a package manager when we could just use pip instead.

If the build developers have any interest in supporting package managers, the bootstrapping procedure must be simplified. If build vendored its dependencies like pip, setuptools, flit, and most low-level build tools, the steps required to install a fully-functional version of build would become:

1. Use installer to install a wheel of itself (How to bootstrap? installer#150)
2. Use installer to install a wheel of build

Consider this a feature request from a package manager down on the front lines. For now, it looks like we'll have to stick with pip.

[rossburton]

FWIW, I wrote and only just stopped maintaining a clone of build at github.com/rossburton/picobuild, which does the bare minimum (no isolation) and does vendor it’s dependencies.

I wrote it because of this problem, but we’ve now moved to 3.11 so it’s no longer a problem for us.

[henryiii]

What's the problem with:

• bootstrap installer (& flit-core)
• Use installer to install wheels of build and all it's dependencies, and all pure Python projects that have wheels, actually

Anything with a pure-Python wheel really can and should be installed via installer rather than rebuilt (literally into an identical wheel) by build, then installed anyway. All of build's dependencies are pure Python wheels, I believe. Having a packaging system flexible enough to choose to start from a wheel for pure Python and from source for non-pure Python would be ideal, I think, and is what we have done in Pyodide, for example, and it's great.

Pretending that dependencies don't exist because they aren't required on Python 3.11 or Linux isn't an option for us.

Some of them simply aren't dependencies, or aren't required during the build process. Also, when you are building in a single environment as a package manager (such as in Spack), you have to include the -x at least some of the time; you simply can't always verify the build dependencies. Some of them (like cmake and ninja) may be listed but don't need the Python package. Others will list a nonsense limit or pin on setuptools or other requirements (numpy has prompted a lot of <60 pinning on setuptools, which is incorrect, it's actually an environment variable that's needed for the rare cases people are using numpy.distutils, and can be set in setup.py). The requirements listed in pyproject.toml are for the PyPI ecosystem, not conda, or spack, or other package managers. It's nice if you can use it, but you simply can't use it everywhere if you are controlling packaging.

Part of the problem is Spack's over-simplified build process, which assumes everything can be built in a single environment. This is simply not always the case. For example, if setuptools_scm is present, the file inclusion system for setuptools changes for all setuptools packages (not just the one that requested it). It could potentially break another package, just by being present. At some point, everything in one environment breaks down.

[adamjstewart]

I agree that Spack's build process is a bit oversimplified and designed more for C/C++/Fortran packages than it is for Python packages. I would like to move more pure-Python packages from sdists to bdists, but the porting process is slow and painful. Ideally I would do this in an automated way, however we also need to be able to support packages without wheels, or packages where there were no wheels, then there were py2.py3 wheels, and now there are py3 wheels and the URL keeps changing. Since Spack packages support multiple versions of the same package, we have to define a single recipe that builds the package with some uniformity. We also may require patches (as is currently the case with build: #266), which make wheels difficult to use.

In the long term, I want to port things just so we can auto-detect dependencies stored in wheel metadata. In the longer term, if PyPI were to publish a downloadable index of packages, versions, and requirements, we may even remove all pure-Python packages from Spack and simply automate those installations. The tricky point is that Spack is heavily used at national labs with strict security requirements and air-gapped networks where we need to create a source cache ahead of time and checksum it once it makes it to the system.

Anyway, this is more of a Spack problem than a build problem, so I won't bore you with the details. I don't think it's impossible to support bootstrapping build if all of its dependencies are installed from wheels anyway, but with Spack's current design this is a bit cumbersome.

[layday]

I don't think the problem is one of (non-)isolation, but simply that of the nature of the package as a distinct entity. For sure, it is annoying to have a bunch of packages with their own unique build requirements, build steps and install steps (flit, installer and pyproject-hooks). But that's a problem which is easily surmountable.

I don't think the process described by Filipe is particularly onerous either. What I think is a bit of a showstopper here is specifying build such that all of its dependencies are installable, some of which are not built with flit, as a prerequisite for installing build. How do you install importlib-metadata or colorama? importlib-metadata needs setuptools, and setuptools needs build to build, but build hasn't been installed yet. Do you clone build with setuptools and run it from path? And what about all of our test and doc deps? Even if they can all be built with flit, you'd probably rather only have to special-case a small number of packages and not everything that may come in build's path.

This is why my suggestion has been to maintain a "build lite" package (it might not even be a package proper - the details are up to you) that can be used internally by your package manager to build Python packages, and expose a separate build package for user consumption. If we can provide a build spec or build script to help with this, I'm onboard.

[adamjstewart]

If we can provide a build spec or build script to help with this, I'm onboard.

If the developers of this project were interested in maintaining a build-lite or build-core package on PyPI, we could definitely use that. Basically identical to picobuild mentioned above but with maintenance.

With the vendoring I suggested above, build and installer can be installed from wheels, and then all of the problems you mentioned with importlib-metadata, colorama, setuptools, flit, etc. would disappear. So the process would be bootstrap build/installer, and then everything else depends on them. This would be the ideal case from my end. Without vendoring, we'll need to change a lot of Spack's packaging model.

[henryiii]

however we also need to be able to support packages without wheels

We are talking about build's dependencies, not arbitrary packages. I'd expect this would be useful for other packages, but it's not a requirement - just moving build's dependencies over to installer would be enough. Several of those dependencies are not changing, like importlib-metadata - it won't even be relevant in a few months. Colorama is not a dependency, I believe we even run our tests without it to verify it is not required. URLs change whenever a version updates, so in the unlikely case you'd need to change "py2.py3" to "py3", it would be done during a version bump. If something decided to become compiled, then you'd have to move back to using build to build it (which is fine, as long as it's not one of our dependencies!)

I think we can resolve the patch issues, and depending on the patch, you probably can apply them to wheels too with a bit of modification. The source files are the same, just the paths change a bit. And, again, it's only a small set of packages (build's dependencies) that need to use installer instead of build. Everything else can temporarily switch to build if patches are needed.

FYI, have no idea if it's useful here at all, but you can build a build.pyz single file that runs without any dependencies:

# noxfile.py
import nox
from pathlib import Path

@nox.session(python="3.7")
def zipapp(session: nox.Session) -> None:
    tmpdir = session.create_tmp()

    # Build a distribution
    session.run(
        "python",
        "-m",
        "pip",
        "install",
        "--no-compile",
        ".",  # this is in build's source, other use "build==<version>" here
        f"--target={tmpdir}",
    )

    # Build the zipapp out of the local directory
    outfile = Path("build.pyz").resolve()
    session.chdir(tmpdir)
    session.run(
        "python",
        "-m",
        "zipapp",
        "--compress",
        "--python=/usr/bin/env python3",
        "--main=build.__main__:entrypoint",
        f"--output={outfile}",
        ".",
    )

Running nox creates build.pyz, and build.pyz is a single file that runs anywhere with all of its dependencies included. Not sure if it's helpful, but I've used things like this occasionally.

[FFY00]

I was replying (hence my missclick of the reopen button above), but then had to go to my flight back from FOSDEM, so some parts of my reply are a bit redundant with what was said above.

---

I would like to once again remind everyone that not only do Python 3.7–3.10 exist, but that we need to continue to support them until at least 2026 when Python 3.10 reaches EOL.

Sorry, I don't want to sound harsh, but we do not. The CPython project commits to supporting older CPython versions in bugfix, and then security mode, for a certain timeline. This has nothing to do with this project.

That said, we do want to try to support older Python versions until they are EOL, but all that is made at a best-effort capability. This project has no funding, and all the maintainers are volunteers, with some maybe getting some time from their employers to work on certain issues, but AFAIK that's almost always limited in scope, and somewhat rare.

So far our track record has been okay I'd say, with us supporting CPython versions that are only in security maintenance mode, or even that have been EOL. Mirroring the CPython maintennance mode would actually be worse.

Additionally, supporting a certain Python version does not mean trade-offs won't be made.

Pretending that dependencies don't exist because they aren't required on Python 3.11 or Linux isn't an option for us.

We are not pretending. Those dependencies are literally not required to use the core functionality of this package. You do not need them during the bootstrapping process. importlib_metadata is only needed in the build.env module, and packaging is only needed when you perform the dependency check.

But the process you've described above is prohibitively difficult, and I can't imagine that even the developers of build use this process to install build. I'm starting to come to the conclusion that the only realistic way to install build is with pip, and that while it is technically possible to bootstrap without pip, this process is neither recommended nor supported.

Yes, because keeping projects like build maintainable, while keeping them easily bootstrapable for everyone is inherently difficult, especially when you lack resources. The bootstrap process for build is s something that I have always kept in mind and tried to simply when writing this project, but improving it is something I definitely something I still want to, but I, and all of the other maintainers, are not machines, we are people with lives and (for some us at least, crappy) things to deal with. Unfortunately, as much I would absolutely love to, time isn't unlimited or free. I have a Github sponsors profile if you'd like to donate something to get me to prioritize this issue for you.

https://github.com/sponsors/FFY00

I also am available for hire as a contractor if you want guaranteed deliverables.

Similarly, some of the other maintainers may also be available in a similar capacity too.

And if pip is required to install build, then there's no reason to use it in a package manager when we could just use pip instead.

I agree with you. If pip is usable for your use-case, then just use it instead. Do not make things more complicated for you.

The reason I started this project was because that is not an option for us in Arch Linux, due to our guideline against vendoring dependencies.

If the build developers have any interest in supporting package managers, the bootstrapping procedure must be simplified. If build vendored its dependencies like pip, setuptools, flit, and most low-level build tools, the steps required to install a fully-functional version of build would become:

...but... this rules out a huge number of package managers (or, rather downstream distributors). I get this works, and makes things easier, for you, but it does the opposite for a huge number of people. And, as we mentioned above, pip already solves this use-case! What you propose actually goes directly against the whole reason I created this project 😅!

Will you attend PyCon US this year, or would you be available to have a remote call? This issue is very complex, so I would love to have a chat with you.

Consider this a feature request from a package manager down on the front lines. For now, it looks like we'll have to stick with pip.

I feel your pain, as I am a packager for a major Linux distribution 😊, and have been for -- checks notes -- almost 8 years... oh god 😳. But please consider, that what you propose will make things significantly harder for others.

Your pain is still valid, and your feedback highly appreciated, but unfortunately there are several other situations that we must consider. I want to find a solution that works as best as possible for everyone.

---

I think it is important to mention that I do not mean to sound harsh or overly negative to you. Text communication sucks at portraying social cues.

There are absolutely no hard feelings, and I do appreciate your feedback, but please try to understand better the reason behind or design decisions, and the use-cases that we have to support, while discussing things like this. I know it's super easy to get caught up in your own issue but that's not very productive. I myself struggle with this, so I know it's hard.

---

Side-note: If you consider the bootstrapping process I described prohibitively difficult, I wonder how you bootstrap GCC 🤣

[FFY00]

If the developers of this project were interested in maintaining a build-lite or build-core package on PyPI, we could definitely use that. Basically identical to picobuild mentioned above but with maintenance.

Why do you need a package? Here's a "build-lite" command.

python -c "import $backend; $backend.build_wheel('.')"

Where $backend is the build backend used.

I haven't looked into Spack's packaging process, but it's possible that it is actually smart enough to generate those commands automatically. If you have something like Fedora's macro system, I could even help you maintain the Python packaging macros.

[adamjstewart]

@FFY00 thanks, that background is actually extremely helpful! I had no idea that Arch Linux forbids vendoring, can you link me to some docs on this? Spack has also been fighting against vendoring for years (PyTorch and TensorFlow are particularly bad about this) so we don't have to patch things 10 different times and we can make air-gapped installs easier. The only time we encourage vendoring is to resolve circular dependencies.

It kinda feels like Spack and Arch Linux are at odds and build is simply caught in the middle. If we ever move away from sdists and prefer bdists, many of the bootstrapping issues I described above would go away. Many of our developers are wary of wheels, but I'm trying to convince them that they aren't that scary.

I definitely feel your pain. I'm just a broke grad student who's been trying to single-handedly maintain the Python ecosystem in Spack for the past 9 years. I'm also a volunteer, so I would never expect or assume that a feature request would imply a feature demand.

P.S. Never been to PyCon but would be down for a Zoom chat sometime 😄