fwupd is attempting to download .gz metadata regardless URL given in conff

[CragW]

Describe the bug
fwupd is not using the MetadataURI url given in the conff file at /etc/fwupd/remotes.d/oem-embargo.conf. The conff has configured MetadataURI=https://fwupd.org/downloads/firmware-hash.xml.xz however in fwupd daemon log it is downloading from gz variant.

Steps to Reproduce
sudo fwupdtool refresh --force

Expected behavior
fwupd is reading the embargo.conf and respect to the MetadataURI for metadata sources.

fwupd version information

$ fwupdmgr --version
compile   org.freedesktop.fwupd         1.9.5
compile   com.hughsie.libxmlb           0.3.10
compile   com.hughsie.libjcat           0.1.9
runtime   org.freedesktop.fwupd-efi     1.4
compile   org.freedesktop.gusb          0.4.5
runtime   com.dell.libsmbios            2.4
runtime   org.freedesktop.gusb          0.4.5
runtime   org.freedesktop.fwupd         1.9.5
runtime   org.kernel                    6.5.0-26-generic

Please note how you installed it (apt, dnf, pacman, source, etc): apt

Additional questions

• Operating system and version: Ubuntu 23.10
• Have you tried rebooting? Yes
• Is this a regression? No

[hughsie]

Is /etc/fwupd/remotes.d/oem-embargo.conf still set as .xz or did fwupd "upgrade" it to gz? If you set it to zst instead does that work as a workaround?

[CragW]

Is /etc/fwupd/remotes.d/oem-embargo.conf still set as .xz or did fwupd "upgrade" it to gz?

The conff persists with xz in file.

If you set it to zst instead does that work as a workaround?

With embargo.conf it goes to gz anyway. However, with lvfs.conf the stable target it takes effect by changing xz to zst.

[CragW]

I've just tested this issue with fwupd 1.9.16, the metadata persist with zst regardless the MetadataURI in conff is given to xz.

It looks like fwupd 1.9.5 persists with gz, while 1.9.16 persists with zst. Hereunder is a small slice from fwupd 1.9.16 runtime log.

$ sudo fwupdtool refresh --force -vv 2>&1
...
12:44:30.541 Fwupd                changing metadata URI from https://fwupd.org/downloads/firmware-hashtag.xml.zst to https://fwupd.org/downloads/firmware-16598-hashtag.xml.zst
12:44:30.541 Fwupd                downloading https://fwupd.org/downloads/firmware-16598-hashtag.xml.zst
...

[superm1]

1.9.16 that makes sense - it matches what we do from 16ad8a9

1.9.5, what version of libxmlb and libjcat do you have compiled against?

[superm1]

I suspect that this is just a bug in older fwupd 1.9.x releases, there shouldn't be anything we can or will do about it.

[CragW]

Sounds like an intentional enforcement to a predefined postfix file format regardless metadatauri, as gz is in older fwupd. If that is the case then this is probably a won't fix issue.

[xvitaly]

It looks like fwupd is broken now on Fedora 39:

$ fwupdmgr refresh
Updating lvfs
Downloading…             [************************************** ]
Successfully downloaded new metadata: 0 local devices supported

It downloads metadata in new format but can't find any supported devices.

$ fwupdmgr --version
compile   org.freedesktop.fwupd         1.9.16
compile   com.hughsie.libxmlb           0.3.17
compile   com.hughsie.libjcat           0.2.1
runtime   org.freedesktop.fwupd-efi     1.4
compile   org.freedesktop.gusb          0.4.8
runtime   com.hughsie.libjcat           0.2.1
runtime   org.freedesktop.gusb          0.4.8
runtime   org.freedesktop.fwupd         1.9.16
runtime   org.kernel                    6.8.4-200.fc39.x86_64

The last working version: fwupd-1.9.15-1.fc39

[superm1]

It looks like fwupd is broken now on Fedora 39:

$ fwupdmgr refresh
Updating lvfs
Downloading…             [************************************** ]
Successfully downloaded new metadata: 0 local devices supported

It downloads metadata in new format but can't find any supported devices.

$ fwupdmgr --version
compile   org.freedesktop.fwupd         1.9.16
compile   com.hughsie.libxmlb           0.3.17
compile   com.hughsie.libjcat           0.2.1
runtime   org.freedesktop.fwupd-efi     1.4
compile   org.freedesktop.gusb          0.4.8
runtime   com.hughsie.libjcat           0.2.1
runtime   org.freedesktop.gusb          0.4.8
runtime   org.freedesktop.fwupd         1.9.16
runtime   org.kernel                    6.8.4-200.fc39.x86_64

The last working version: fwupd-1.9.15-1.fc39

Might be same issue as #7060

[hughsie]

Successfully downloaded new metadata: 0 local devices supported

@xvitaly Grab the new libxmlb in updates-testing please.

[hughsie]

@CragW on 1.9.5 what does sudo fwupdtool refresh -vv say when the metadata ends with xz?

[xvitaly]

Grab the new libxmlb in updates-testing please.

It works now. Thanks.

Successfully downloaded new metadata: 5 local devices supported

[CragW]

@CragW on 1.9.5 what does sudo fwupdtool refresh -vv say when the metadata ends with xz?

Hereunder is the embargo.conf under /etc/fwupd/remotes.d/, I've masked the hash with x and the same in the txt output.

[fwupd Remote]
Enabled=true
Title=Embargoed for dell
MetadataURI=https://fwupd.org/downloads/firmware-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xml.xz
ReportURI=https://fwupd.org/lvfs/firmware/report
OrderBefore=lvfs,fwupd
AutomaticReports=true

ubuntu-fwupd-1.9.5-refresh-xz.txt

[hughsie]

Okay, weird -- I can't reproduce the .xz to gz thing at all with the proper dell embargo remote and 1.9.5.

[superm1]

Libxmlb version?

[CragW]

Libxmlb version?

Sorry missed this question, the runtime libxmlb is 0.3.10.

$ dpkg -l | grep libxmlb
ii  gir1.2-xmlb-2.0:amd64                         0.3.10-2                                amd64        GObject introspection data for libxmlb
ii  libxmlb-dev:amd64                             0.3.10-2                                amd64        Development files for libxmlb
ii  libxmlb2:amd64                                0.3.10-2                                amd64        Binary XML library