qubes-builder-v2 - Build archlinux packages - Executor qubes - Crash

[neowutran]

Brief summary

I am unable to build the archlinux packages using qubes-builder-v2.
Docker executor crash
Local executor is not supported
Qubes executor crash

This issue is specific to the qubes executor

Steps to reproduce

sudo ./qb package fetch prep build

Expected behavior

The archlinux packages have been build

Actual behavior

The builder crash (multiple reasons, missing packages + missing binary qubes-x86_64-build )

Patch I applied to myself to make it work:

diff --git a/qubesbuilder/plugins/build_archlinux/__init__.py b/qubesbuilder/plugins/build_archlinux/__init__.py
index c14598e..8fd8954 100644
--- a/qubesbuilder/plugins/build_archlinux/__init__.py
+++ b/qubesbuilder/plugins/build_archlinux/__init__.py
@@ -302,6 +302,7 @@ class ArchlinuxBuildPlugin(ArchlinuxDistributionPlugin, BuildPlugin):
                     "sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d",
                     "sudo pacman-key --init",
                     "sudo pacman-key --populate",
+                    "sudo pacman -S --noconfirm python-distutils-extra libxdamage"
                 ]
             else:
                 log.info(
@@ -350,7 +351,7 @@ class ArchlinuxBuildPlugin(ArchlinuxDistributionPlugin, BuildPlugin):
                 f"--syncdeps --noconfirm --skipinteg",
             ]
 
-            cmd += [f"cd {source_dir}", " ".join(build_command)]
+            cmd += [f"cd {source_dir}", f"sudo ln -s /usr/local/bin/archbuild /usr/local/bin/qubes-x86_64-build", " ".join(build_command)]
 
             try:
                 executor.run(
diff --git a/qubesbuilder/plugins/build_archlinux/scripts/update-local-repo.sh b/qubesbuilder/plugins/build_archlinux/scripts/update-local-repo.sh
index 3b7fe06..c58171e 100755
--- a/qubesbuilder/plugins/build_archlinux/scripts/update-local-repo.sh
+++ b/qubesbuilder/plugins/build_archlinux/scripts/update-local-repo.sh
@@ -54,3 +54,7 @@ chroot_cmd /bin/sh -c \
 # Now update system
 chroot_cmd /bin/sh -c \
     "http_proxy='${REPO_PROXY}' pacman -Syu --noconfirm"
+
+chroot_cmd /bin/sh -c \
+    "http_proxy='${REPO_PROXY}' pacman -S --noconfirm python-distutils-extra libxdamage"
+

One of the pacman command I added is indeed useless, I just didn't wanted to spend time thinking where would it be best to run it.
And the proper fix would probably add "python-distutils-extra libxdamage" to a list of packages that need to be installed to compile the thing instead of being a separate pacman call.

---

Building the archlinux iso fail too, I didn't investigated the crash since I don't really needed to rebuild the iso

[fepitre]

What do you mean by archlinux ISO? There is no such thing in Qubes. Also, without exact logs of what's missing it's hard to make any diagnostic. Also, missing qubes-x86_64-build is probably because you have not setup it in your docker image or your dispvm like here: https://github.com/QubesOS/qubes-builderv2/blob/main/dockerfiles/fedora-mock.Dockerfile#L16.

[fepitre]

It means also that an update of the documentation needs to be done too.

[neowutran]

Sorry, I was a bit tired and I didn't used the correct names.
I means archlinux template, not archlinux ISO.

I read the readme multiple times and I was able to build the qubes packages, qubesos iso, and standard qubes os template for quite a long time. But I tend to have some difficulties with building archlinux packages and archlinux template, both using the docker executor or the qubes executor.

Below some of the output I get with the docker executor to build archlinux packages:

19:20:44,201 [executor:docker:80e17673b43e] output: error: guile: missing required signature
19:20:44,201 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/gc-8.2.6-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,201 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,201 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/guile-3.0.9-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,207 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n] error: make: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: patch: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: pkgconf: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: sudo: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: texinfo: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: which: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: error: base-devel: missing required signature
19:20:44,207 [executor:docker:80e17673b43e] output: 
19:20:44,207 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/make-4.4.1-2-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,207 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,207 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/patch-2.7.6-10-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,207 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,207 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/pkgconf-2.1.1-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,207 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,207 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/sudo-1.9.15.p5-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,207 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,207 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/texinfo-7.1-2-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,208 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,208 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/which-2.21-6-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,208 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n]
19:20:44,208 [executor:docker:80e17673b43e] output: :: File /var/cache/pacman/pkg/base-devel-1-1-any.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
19:20:44,208 [executor:docker:80e17673b43e] output: Do you want to delete it? [Y/n] error: failed to commit transaction (invalid or corrupted package (PGP signature))
19:20:44,228 [executor:docker:80e17673b43e] output: 
19:20:44,228 [executor:docker:80e17673b43e] output: Errors occurred, no packages were upgraded.
19:20:44,237 [executor:docker:80e17673b43e] output: ==> ERROR: Failed to install packages to new root
19:20:44,240 [executor:docker:80e17673b43e] output: ==> ERROR: Failed to install all packages
19:20:44,245 [executor:docker:80e17673b43e] output: tput: unknown terminal "unknown"
Error: vmm-xen:vm-archlinux-rolling.x86_64:archlinux: Failed to build PKGs: Failed to run 'sudo mkdir -p -- /builder /builder/build /builder/plugins /builder/distfiles&&sudo chown -R -- user:user /builder&&sudo cp /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /usr/local/share/devtools/makepkg.conf.d/qubes-x86_64.conf&&sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf&&sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d&&sudo pacman-key --init&&sudo pacman-key --populate&&sudo mkdir -p /builder/cache/qubes-x86_64&&sudo mkarchroot -C /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf -M /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /builder/cache/qubes-x86_64/root base-devel&&sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf --enable-builder-local&&sudo /builder/plugins/build_archlinux/scripts/update-local-repo.sh /builder/cache/qubes-x86_64/root /builder/repository&&mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz /builder/vmm-xen&&mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz.sig /builder/vmm-xen&&cd /builder/vmm-xen&&sudo qubes-x86_64-build -r /builder/cache --  -d /builder/repository:/builder/repository --  --syncdeps --noconfirm --skipinteg' (status=255)..
Traceback (most recent call last):
  File "/home/user/qubes-builderv2/qubesbuilder/plugins/build_archlinux/__init__.py", line 356, in run
    executor.run(
  File "/home/user/qubes-builderv2/qubesbuilder/executors/container.py", line 239, in run
    raise ExecutorError(f"Failed to run '{final_cmd}' (status={rc}).")
qubesbuilder.executors.ExecutorError: Failed to run 'sudo mkdir -p -- /builder /builder/build /builder/plugins /builder/distfiles&&sudo chown -R -- user:user /builder&&sudo cp /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /usr/local/share/devtools/makepkg.conf.d/qubes-x86_64.conf&&sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf&&sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d&&sudo pacman-key --init&&sudo pacman-key --populate&&sudo mkdir -p /builder/cache/qubes-x86_64&&sudo mkarchroot -C /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf -M /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /builder/cache/qubes-x86_64/root base-devel&&sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf --enable-builder-local&&sudo /builder/plugins/build_archlinux/scripts/update-local-repo.sh /builder/cache/qubes-x86_64/root /builder/repository&&mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz /builder/vmm-xen&&mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz.sig /builder/vmm-xen&&cd /builder/vmm-xen&&sudo qubes-x86_64-build -r /builder/cache --  -d /builder/repository:/builder/repository --  --syncdeps --noconfirm --skipinteg' (status=255).

Below some of the output I get with the qubes executor to build archlinux packages:

19:12:44,560 [executor:qubes:disp1239] output: sudo: qubes-x86_64-build: command not found
Error: vmm-xen:vm-archlinux-rolling.x86_64:archlinux: Failed to build PKGs: Failed to run '/usr/bin/qvm-run-vm -- disp1239 env -- VERBOSE=1 DEBUG=1 BACKEND_VMM=xen DIST=archlinux PACKAGE_SET=vm bash -c 'sudo cp /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /usr/local/share/devtools/makepkg.conf.d/qubes-x86_64.conf && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf && sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d && sudo pacman-key --init && sudo pacman-key --populate && sudo mkdir -p /builder/cache/qubes-x86_64 && sudo mkarchroot -C /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf -M /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /builder/cache/qubes-x86_64/root base-devel && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf --enable-builder-local && sudo /builder/plugins/build_archlinux/scripts/update-local-repo.sh /builder/cache/qubes-x86_64/root /builder/repository && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz /builder/vmm-xen && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz.sig /builder/vmm-xen && cd /builder/vmm-xen && sudo qubes-x86_64-build -r /builder/cache --  -d /builder/repository:/builder/repository --  --syncdeps --noconfirm --skipinteg'' (status=1)..
Traceback (most recent call last):
  File "/home/user/qubes-builderv2/qubesbuilder/plugins/build_archlinux/__init__.py", line 356, in run
    executor.run(
  File "/home/user/qubes-builderv2/qubesbuilder/executors/qubes.py", line 322, in run
    raise ExecutorError(
qubesbuilder.executors.ExecutorError: Failed to run '/usr/bin/qvm-run-vm -- disp1239 env -- VERBOSE=1 DEBUG=1 BACKEND_VMM=xen DIST=archlinux PACKAGE_SET=vm bash -c 'sudo cp /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /usr/local/share/devtools/makepkg.conf.d/qubes-x86_64.conf && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf && sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d && sudo pacman-key --init && sudo pacman-key --populate && sudo mkdir -p /builder/cache/qubes-x86_64 && sudo mkarchroot -C /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf -M /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /builder/cache/qubes-x86_64/root base-devel && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf --enable-builder-local && sudo /builder/plugins/build_archlinux/scripts/update-local-repo.sh /builder/cache/qubes-x86_64/root /builder/repository && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz /builder/vmm-xen && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz.sig /builder/vmm-xen && cd /builder/vmm-xen && sudo qubes-x86_64-build -r /builder/cache --  -d /builder/repository:/builder/repository --  --syncdeps --noconfirm --skipinteg'' (status=1).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_base.py", line 65, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1719, in invoke
    rv.append(sub_ctx.command.invoke(sub_ctx))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_package.py", line 86, in build
    _component_stage(
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_package.py", line 40, in _component_stage
    p.run(stage=stage_name)
  File "/home/user/qubes-builderv2/qubesbuilder/plugins/build_archlinux/__init__.py", line 365, in run
    raise BuildError(msg) from e
qubesbuilder.plugins.build.BuildError: vmm-xen:vm-archlinux-rolling.x86_64:archlinux: Failed to build PKGs: Failed to run '/usr/bin/qvm-run-vm -- disp1239 env -- VERBOSE=1 DEBUG=1 BACKEND_VMM=xen DIST=archlinux PACKAGE_SET=vm bash -c 'sudo cp /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /usr/local/share/devtools/makepkg.conf.d/qubes-x86_64.conf && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf && sudo rm -rf /etc/pacman.d/gnupg/private-keys-v1.d && sudo pacman-key --init && sudo pacman-key --populate && sudo mkdir -p /builder/cache/qubes-x86_64 && sudo mkarchroot -C /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf -M /builder/plugins/chroot_archlinux/conf/makepkg-x86_64.conf /builder/cache/qubes-x86_64/root base-devel && sudo /builder/plugins/chroot_archlinux/scripts/generate-pacman /builder/plugins/chroot_archlinux/conf/pacman.conf.j2 /usr/local/share/devtools/pacman.conf.d/qubes-x86_64.conf --enable-builder-local && sudo /builder/plugins/build_archlinux/scripts/update-local-repo.sh /builder/cache/qubes-x86_64/root /builder/repository && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz /builder/vmm-xen && mv /builder/distfiles/vmm-xen/xen-4.17.4.tar.gz.sig /builder/vmm-xen && cd /builder/vmm-xen && sudo qubes-x86_64-build -r /builder/cache --  -d /builder/repository:/builder/repository --  --syncdeps --noconfirm --skipinteg'' (status=1)..

( I am not really familiar with docker so I preferred to search and find a solution for the qubes executor as show in my first comment. )

The builder.yml I am using is:

git:
  baseurl: https://github.com
  prefix: qubesos/qubes-
  branch: release4.2
  maintainers:
    # marmarek
    - '0064428F455451B3EBE78A7F063938BA42CFA724'
    # simon
    - '274E12AB03F2FE293765FC06DA0434BC706E1FCF'

backend-vmm: xen
debug: true
verbose: true
qubes-release: r4.2
gpg-client: qubes-gpg-client-wrapper

sign-key:
  rpm: 77EEEF6D0386962AEA8CF84A9B8273F80AC219E6
  deb: 77EEEF6D0386962AEA8CF84A9B8273F80AC219E6
  archlinux: 77EEEF6D0386962AEA8CF84A9B8273F80AC219E6
  iso: 77EEEF6D0386962AEA8CF84A9B8273F80AC219E6

cache:
  vm-archlinux:
    packages:
      - bin86
      - bridge-utils
      - conntrack-tools
      - dconf
      - desktop-file-utils
      - dev86
      - fakeroot
      - gawk
      - gcc
      - git
      - gnome-packagekit
      - graphicsmagick
      - haveged
      - iasl
      - iproute2
      - iptables
      - librsvg
      - libsystemd
      - libutil-linux
      - libx11
      - lsb-release
      - lzo
      - make
      - make
      - networkmanager
      - nftables
      - notification-daemon
      - ntp
      - openssl
      - pacman-contrib
      - pandoc
      - parted
      - patch
      - pixman
      - pkg-config
      - procps-ng
      - python
      - python-dbus
      - python-gobject
      - python-lxml
      - python-setuptools
      - python-xdg
      - sed
      - shared-mime-info
      - socat
      - tinyproxy
      - wget
      - xdg-utils
      - yajl
      - zenity
      - blas
      - cblas
      - lapack
      - libimagequant
      - libraqm
      - openjpeg2
      - python-cairo
      - python-numpy
      - python-pillow

distributions:
  - vm-archlinux


insecure-skip-checking:
 - gui-agent-linux
 - builder-archlinux
 - core-agent-linux
# - meta-packages


templates:
  - archlinux:
      dist: archlinux

components:
  - builder-archlinux:
      packages: False
      branch: master
#      url: https://github.com/neowutran/qubes-builder-archlinux
  - vmm-xen
  - core-vchan-xen
  - core-qubesdb
  - core-qrexec
  - linux-utils
  - core-agent-linux
#     url: https://github.com/neowutran/qubes-core-agent-linux
#     branch: allowpython3.13
  - gui-common
  - gui-agent-linux
#     branch: archlinuxpipewire
#     url: https://github.com/neowutran/qubes-gui-agent-linux
  - app-linux-split-gpg:
     branch: main
  - app-linux-input-proxy:
     branch: main
  - app-linux-usb-proxy:
     branch: main
  - meta-packages
#     branch: arch
#     url: https://github.com/neowutran/qubes-meta-packages

less-secure-signed-commits-sufficient:
  - vmm-xen
  - core-vchan-xen
  - core-qubesdb
  - core-qrexec
  - linux-utils
#  - core-agent-linux
  - gui-common
#  - gui-agent-linux
  - app-linux-split-gpg
  - app-linux-input-proxy
  - app-linux-usb-proxy
  - meta-packages

#executor:
# type: docker
# options:
#   image: "qubes-builder-fedora:latest"
#executor: 
#  type: local
executor:
  type: qubes
  options:
    dispvm: "qubes-builder-dvm"

stages:
  - fetch
  - prep
  - build
  - post
  - verify
  - sign:
      executor:
        type: local
  - publish:
      executor:
        type: local
  - upload:
      executor:
        type: local

and I build the packages with that command:

sudo ./qb package fetch prep build

---

( For the qubes executor, I indeed didn't created manually the file "qubes-x86_64-build" in the dispvm. If it need to be manually created, I didn't found a reference to that in the readme. But probably better to do that automatically in the build script )