qubes-qrexec-policy-daemon rule matching breaks/works by seemingly unrelated filesystem actions

[aronowski]

How to file a helpful issue

Qubes OS release

4.2.1

Brief summary

Weird/counterintuitive behavior of qubes-qrexec-policy-daemon, inability to find matching rules until a service restart or seemingly-unrelated filesystem manipulations take place.

Steps to reproduce

Install Qubes OS 4.2.1 preferably inside a virtual machine like described here to help with debugging the issue by rewinding time to "last known bad/good condition". As part of such virtualized installation, sys-usb may be disabled and a full control of USB devices handled to dom0.

Then attempt to configure the installed system to work with qubes-builderv2 with the following commands:

[user@dom0 ~]$ qvm-run --pass-io fedora-39-xfce -- sudo dnf install -y python3-packaging python3-click python3-lxml createrepo_c devscripts gpg python3-pyyaml rpm docker python3-docker podman python3-podman reprepro python3-pathspec rpm-sign rb_libtorrent-examples openssl tree mock python3-jinja2-cli pacman m4 asciidoc rsync sequoia-sq sequoia-sqv sequoia-chameleon-gnupg qubes-gpg-split dnf-plugins-core createrepo_c debootstrap devscripts dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA python3-pyyaml python3-sh rpm-build rpmdevtools wget python3-debian reprepro systemd-udev
[user@dom0 ~]$ qvm-shutdown fedora-39-xfce

[user@dom0 ~]$ qvm-create -l yellow -t fedora-39-xfce work-qubesos
[user@dom0 ~]$ qvm-create -l red -t fedora-39-xfce qubes-builder-dvm
[user@dom0 ~]$ qvm-prefs qubes-builder-dvm template_for_dispvms True
[user@dom0 ~]$ qvm-prefs work-qubesos default_dispvm qubes-builder-dvm

[user@dom0 ~]$ qvm-run --pass-io work-qubesos -- git clone https://github.com/qubesos/qubes-builderv2

[user@dom0 ~]$ qvm-run --pass-io work-qubesos 'cat /home/user/qubes-builderv2/rpc/policy/50-qubesbuilder.policy' > /home/user/50-qubesbuilder.policy

[user@dom0 ~]$ sudo mv /home/user/50-qubesbuilder.policy /etc/qubes/policy.d/

Try to build the fedora-39 template - qubes-qrexec-policy-daemon should fail to find an appropriate rule:

[user@dom0 ~]$ qvm-run --pass-io work-qubesos -- bash -c "cd /home/user/qubes-builderv2/ && ./qb --builder-conf=./example-configs/qubes-os-r4.2.yml -t fedora-39 template fetch prep"
Running template stage: fetch
Running template stage: prep
Error: fedora-39: Failed to prepare template.
Traceback (most recent call last):
  File "/home/user/qubes-builderv2/qubesbuilder/plugins/template/__init__.py", line 587, in run
    executor.run(
  File "/home/user/qubes-builderv2/qubesbuilder/executors/qubes.py", line 165, in run
    raise ExecutorError("Failed to create disposable qube")
qubesbuilder.executors.ExecutorError: Failed to create disposable qube

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_base.py", line 65, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    rv.append(sub_ctx.command.invoke(sub_ctx))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_template.py", line 73, in prep
    _template_stage(
  File "/home/user/qubes-builderv2/qubesbuilder/cli/cli_template.py", line 36, in _template_stage
    p.run(stage=stage_name, template_timestamp=template_timestamp)
  File "/home/user/qubes-builderv2/qubesbuilder/plugins/template/__init__.py", line 596, in run
    raise TemplateError(msg) from e
qubesbuilder.plugins.template.TemplateError: fedora-39: Failed to prepare template.

Further proof of the inability to find the appropriate rule:

[user@dom0 ~]$ journalctl --grep=admin.vm.CreateDisposable --lines=1 --no-pager -u qubes-qrexec-policy-daemon 
May 03 16:02:08 dom0 qrexec-policy-daemon[1542]: qrexec: admin.vm.CreateDisposable: work-qubesos -> qubes-builder-dvm: denied: no matching rule found

Change the policy file's filesystem owner, group and permissions as suggested here:

[user@dom0 ~]$ sudo chown root:qubes /etc/qubes/policy.d/50-qubesbuilder.policy
[user@dom0 ~]$ sudo chmod 664 /etc/qubes/policy.d/50-qubesbuilder.policy

however, the same exception should be thrown (if that's reproducible, the aforementioned PR may be closed - however, please read further). Take a live snapshot of the virtual machine hosting Qubes OS right here and now to rewind time in case of unsuccessful experiments.

From this point, each of the actions described below happens from that snapshot being restored.

Restarting qubes-qrexec-policy-daemon (or rebooting Qubes OS - same thing) should fix the problem:

[user@dom0 ~]$ sudo systemctl restart qubes-qrexec-policy-daemon

These commands should also fix the problem:

[user@dom0 ~]$ sudo cp /etc/qubes/policy.d/50-qubesbuilder.policy /home/user/
[user@dom0 ~]$ sudo rm -f /etc/qubes/policy.d/50-qubesbuilder.policy 
[user@dom0 ~]$ sudo cp /home/user/50-qubesbuilder.policy /etc/qubes/policy.d/

Or these:

[user@dom0 ~]$ ls -i /etc/qubes/policy.d/50-qubesbuilder.policy 
674362 /etc/qubes/policy.d/50-qubesbuilder.policy
[user@dom0 ~]$ cp /etc/qubes/policy.d/50-qubesbuilder.policy /home/user/
[user@dom0 ~]$ ls -i /home/user/50-qubesbuilder.policy 
674360 /home/user/50-qubesbuilder.policy
[user@dom0 ~]$ sudo cp /home/user/50-qubesbuilder.policy /etc/qubes/policy.d/
[user@dom0 ~]$ ls -i /etc/qubes/policy.d/50-qubesbuilder.policy 
674362 /etc/qubes/policy.d/50-qubesbuilder.policy

---

What does not work:

[user@dom0 ~]$ sudo mv /etc/qubes/policy.d/50-qubesbuilder.policy /home/user/
[user@dom0 ~]$ sudo mv /home/user/50-qubesbuilder.policy /etc/qubes/policy.d/

or

[user@dom0 ~]$ ls -i /etc/qubes/policy.d/50-qubesbuilder.policy 
674362 /etc/qubes/policy.d/50-qubesbuilder.policy
[user@dom0 ~]$ cp /etc/qubes/policy.d/50-qubesbuilder.policy /home/user/
[user@dom0 ~]$ ls -i /home/user/50-qubesbuilder.policy 
674360 /home/user/50-qubesbuilder.policy
[user@dom0 ~]$ sudo mv /home/user/50-qubesbuilder.policy /etc/qubes/policy.d/
[user@dom0 ~]$ ls -i /etc/qubes/policy.d/50-qubesbuilder.policy 
674360 /etc/qubes/policy.d/50-qubesbuilder.policy

Expected behavior

Consistent and intuitive behavior of qubes-qrexec-policy-daemon, rather than relying on unknown/random/undocumented behavior to make it work as intended. Alternatively a clear mention in the documentation that it's preferable to restart the service, which handles it, each time a policy is to be applied.

Actual behavior

Doing seemingly random/unrelated actions related to moving, copying or modifying filesystem characteristics of a file makes qubes-qrexec-policy-daemon work after it fails to handle the policies in that file, which are added the expected way.