xen-balloon consumes 15-30%+ CPU on idle in user qubes

[jamke]

Qubes OS release

R4.1
Used template is based on fedora-38-minimal.
Both OS and template are up to date.

Brief summary

xen-balloon command consumes 30%+ CPU on idle.
Happens with several qubes more often that others based on the same template.

Steps to reproduce

Use qube A while qube B shows no applications and should idle and do nothing.

Expected behavior

Qube B consumes 0% CPU

Actual behavior

Qube B consumes 15-35% CPU constantly or from time to time.
top shows that xen-balloon command is the only one that consumes this.

Additional

• This problem of xen-balloon consuming CPU already happened in past but for dom0 (in 2021-2022):
  https://forum.qubes-os.org/t/high-dom0-cpu-usage-by-xen-balloon/6732/
  But I have everything up to date for R4.1, so it is expected to be fixed long time ago. i also have it in AppVMs, not dom0. Probably related, though.

• It's probably a regression of the last year or so, before that I have never had this situation before, even with the same template.

[UndeadDevel]

I've seen this symptom (~15-20% CPU usage in idle qubes) as well quite often (not sure about 4.1, but definitely in 4.2), but every time I launch a terminal in them and execute top, the CPU usage drops to very low (normal) levels again...did you use an admin console from a management disp VM to "catch" the xen-balloon process? The one thing that sometimes flashed for a moment with high-ish CPU usage in top was the Xorg process, but that may just be related to the fact that I just launched a terminal in that qube.

[jamke]

@UndeadDevel I use terminal (konsole), and top. Nothing fancy in my case. xen-balloon is shown as the top consumer. Can go to 0% in a few seconds, or can stay 15-35% for a longer time, for like minutes.