OCPBUGS-33248: daemon: upgrade os image from local container storage if it exists

[hexfusion]

This PR add the ability to upgrade an os image if it has already been pulled into local container storage. This functionality is behind PinnedImage feature gate. The advantage to this path is that before this change skopeo by default will make a call to the registry to pull the image during upgrade. That does not work for disconnected clusters.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hexfusion
Once this PR has been reviewed and has the lgtm label, please assign djoshy for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@hexfusion: This pull request references Jira Issue OCPBUGS-33248, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This PR add the ability to upgrade an os image if it has already been pulled into local container storage. This functionality is gated behind PinnedImage gates.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@hexfusion: This pull request references Jira Issue OCPBUGS-33248, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[hexfusion]

/jira refresh

[openshift-ci-robot]

@hexfusion: This pull request references Jira Issue OCPBUGS-33248, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @siserafin

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@hexfusion: This pull request references Jira Issue OCPBUGS-33248, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @siserafin

In response to this:

This PR add the ability to upgrade an os image if it has already been pulled into local container storage. This functionality is gated behind PinnedImage gates. The advantage to this path is that prior to this change skopeo by default will make a call to the registry to pull the image during upgrade. That does not work for disconnected clusters.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@hexfusion: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-hypershift	65324b1	link	true	/test e2e-hypershift

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[siserafin]

Verified in a disconnected cluster.

1. Mirror an os image
2. Pull the image from the node
3. Add the firewall rule to block the pulling from registry
   sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
4. Trigger the os upgrade

rpm-ostree --experimental rebase ostree-unverified-image:containers-storage:registry.build03.ci.openshift.org/ci-ln-gv2713b/stable@sha256:fda7f7db7bed8d6691acc2dd9870d492995b6bae6354b17edb3537f2637e113a
Pulling manifest: ostree-unverified-image:containers-storage:registry.build03.ci.openshift.org/ci-ln-gv2713b/stable@sha256:fda7f7db7bed8d6691acc2dd9870d492995b6bae6354b17edb3537f2637e113a
Importing: ostree-unverified-image:containers-storage:registry.build03.ci.openshift.org/ci-ln-gv2713b/stable@sha256:fda7f7db7bed8d6691acc2dd9870d492995b6bae6354b17edb3537f2637e113a (digest: sha256:fda7f7db7bed8d6691acc2dd9870d492995b6bae6354b17edb3537f2637e113a)
ostree chunk layers already present: 36
ostree chunk layers needed: 15 (284.5?MB)
⠁ Staging deployment...                                                                                                                                                                                                                                                                                                                                                                               Staging deployment... done
Downgraded:
  container-selinux 3:2.231.0-1.rhaos4.16.el9 -> 3:2.229.0-1.rhaos4.16.el9
  cri-o 1.29.4-6.rhaos4.16.git0e93ae2.el9 -> 1.29.4-5.rhaos4.16.gitec19fc9.el9
  openshift-kubelet 4.16.0-202405062116.p0.g681e46b.assembly.stream.el9 -> 4.16.0-202404251943.p0.gd1ec84a.assembly.stream.el9
Changes queued for next boot. Run "systemctl reboot" to start a reboot

/label qe-approved