Bootloader ISO EFI Support #13123

klaases · 2021-12-07T23:57:38Z

This PR is an important prerequisite for building minikube ISO on ARM64.

Note: the containerd version has been lowered 1.4.9, as 1.5.8 has a rotating hash issue that prevents the ISO from building properly. For more information, see: #13193

Note: make sure to merge #12892 before submission. [CONFIRMED]

Build the ISO, see https://minikube.sigs.k8s.io/docs/contrib/building/iso/

Use mk ssh to ssh into minikube ISO.

Run [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

Before:

BIOS

After:

UEFI

Have tested the new ISO with the following commands.

Download the latest ISO, and place in minikube/out folder.
curl -LO https://storage.googleapis.com/minikube-builds/iso/13123/minikube-v1.24.0-1640194376-13123.iso

Alternative URL:
gs://minikube-builds/iso/13123/minikube-v1.24.0-1640194376-13123.iso

Run make from minikube home folder.

Run ./out/minikube start --iso-url=file://$(pwd)/out/minikube-v1.24.0-1640194376-13123.iso to start minikube with the new ISO.

Use mk ssh to ssh into minikube ISO.

Run [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS to confirm that minikube is running with the UEFI bootloader.

k8s-ci-robot · 2021-12-07T23:57:48Z

Hi @klaases. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

klaases · 2021-12-07T23:57:51Z

/assign

klaases · 2021-12-07T23:57:56Z

wip: add efi bootloader

minikube-bot · 2021-12-08T00:03:03Z

Can one of the admins verify this patch?

klaases · 2021-12-08T19:35:47Z

This configuration resulted in the following error when building ISO:

g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1116: recipe for target 'gimple-match.o' failed
make[4]: *** [gimple-match.o] Error 4
make[4]: *** Waiting for unfinished jobs....
rm gcc.pod
make[4]: Leaving directory '/mnt/out/buildroot/output/build/host-gcc-initial-9.4.0/build/gcc'
Makefile:4300: recipe for target 'all-gcc' failed
make[3]: *** [all-gcc] Error 2
make[3]: Leaving directory '/mnt/out/buildroot/output/build/host-gcc-initial-9.4.0/build'
package/pkg-generic.mk:247: recipe for target '/mnt/out/buildroot/output/build/host-gcc-initial-9.4.0/.stamp_built' failed
make[2]: *** [/mnt/out/buildroot/output/build/host-gcc-initial-9.4.0/.stamp_built] Error 2
make[2]: Leaving directory '/mnt/out/buildroot'
make[1]: *** [minikube_iso] Error 2
Makefile:286: recipe for target 'minikube_iso' failed
make[1]: Leaving directory '/mnt'
make: *** [out/minikube.iso] Error 2
Makefile:312: recipe for target 'out/minikube.iso' failed
make: *** [out/minikube.iso] Error 2

klaases · 2021-12-10T00:15:06Z

Added additional files for testing.

minikube-bot · 2021-12-21T00:10:38Z

Hi @klaases, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13123/035151d/iso_build.txt

sharifelgamal · 2021-12-22T00:38:02Z

ok-to-build-iso

minikube-bot · 2021-12-22T02:41:50Z

Hi @klaases, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13123/035151d/iso_build.txt

klaases · 2021-12-22T03:11:48Z

Trying to build on a new ubuntu server, rather than trying with Docker.

Trying again with mkisofs installed on new server.

18:41:49 /home/jenkins/go/src/k8s.io/minikube/deploy/iso/minikube-iso/board/iso/x86_64/post-image.sh: 31: /home/jenkins/go/src/k8s.io/minikube/deploy/iso/minikube-iso/board/iso/x86_64/post-image.sh: mkisofs: not found

klaases · 2021-12-22T03:12:03Z

ok-to-build-iso

minikube-bot · 2021-12-22T03:22:15Z

Hi @klaases, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13123/035151d/iso_build.txt

sharifelgamal · 2021-12-22T03:23:44Z

ok-to-build-iso

minikube-bot · 2021-12-22T04:34:03Z

Hi @klaases, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13123/035151d/iso_build.txt

klaases · 2021-12-22T05:14:24Z

Nest steps will be to install gsutil:

https://cloud.google.com/storage/docs/gsutil_install#deb

make: gsutil: Command not found
make: *** [release-iso] Error 127

sharifelgamal · 2021-12-22T07:59:53Z

ok-to-build-iso

minikube-bot · 2021-12-22T09:08:31Z

Hi @klaases, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13123/035151d/iso_build.txt

sharifelgamal · 2021-12-22T17:28:36Z

ok-to-build-iso

minikube-bot · 2021-12-22T18:45:00Z

Hi @klaases, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

sharifelgamal · 2021-12-22T19:16:32Z

/ok-to-test

klaases · 2021-12-22T19:28:20Z

Success - have tested the new ISO with the following commands.

Download the latest ISO, and place in minikube/out folder.
curl -LO https://storage.googleapis.com/minikube-builds/iso/13123/minikube-v1.24.0-1640194376-13123.iso

Alternative URL:
gs://minikube-builds/iso/13123/minikube-v1.24.0-1640194376-13123.iso

Run make from minikube home folder.

Run ./out/minikube start --iso-url=file://$(pwd)/out/minikube-v1.24.0-1640194376-13123.iso to start minikube with the new ISO.

Use mk ssh to ssh into minikube ISO.

Run [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS to confirm that minikube is running with the UEFI bootloader.

minikube-pr-bot · 2021-12-22T20:05:42Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 46.8s    | 44.9s               |
| enable ingress | 28.8s    | 29.2s               |
+----------------+----------+---------------------+

Times for minikube start: 45.0s 43.9s 43.6s 57.6s 43.8s
Times for minikube (PR 13123) start: 45.0s 44.8s 44.1s 45.4s 45.0s

Times for minikube (PR 13123) ingress: 29.1s 29.5s 29.1s 29.2s 29.1s
Times for minikube ingress: 30.0s 25.5s 30.5s 29.0s 29.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 27.4s    | 26.1s               |
| enable ingress | 23.2s    | 22.4s               |
+----------------+----------+---------------------+

Times for minikube start: 30.2s 25.9s 29.3s 25.8s 26.0s
Times for minikube (PR 13123) start: 26.6s 25.8s 25.8s 26.1s 26.4s

Times for minikube (PR 13123) ingress: 20.9s 23.0s 22.9s 22.9s 22.4s
Times for minikube ingress: 25.9s 22.4s 21.9s 22.9s 22.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 40.6s    | 43.8s               |
| enable ingress | 29.8s    | 20.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 23.4s 33.4s 19.4s 53.9s 18.9s
Times for minikube (PR 13123) ingress: 18.9s 18.9s 19.4s 22.9s 23.4s

Times for minikube start: 31.1s 41.4s 44.7s 41.4s 44.4s
Times for minikube (PR 13123) start: 46.3s 45.2s 44.9s 41.3s 41.0s

afbjorklund · 2021-12-22T22:07:42Z

@klaases we can upgrade containerd to 1.4.12, with #13195

that would also address the security issue (CVE-2021-41190)

minikube-pr-bot · 2021-12-22T22:20:40Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyperkit_macOS	TestAddons/Setup (gopogh)	0.00 (chart)
Hyperkit_macOS	TestErrorSpam/setup (gopogh)	0.00 (chart)
Hyperkit_macOS	TestFunctional/serial/AuditLog (gopogh)	0.00 (chart)
Hyperkit_macOS	TestFunctional/serial/StartWithProxy (gopogh)	0.00 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/pause/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/pause/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/parallel/DistinctCurrentSteps (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/parallel/IncreasingCurrentSteps (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/unpause/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/unpause/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMountStart/serial/StartWithMountFirst (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/CopyFile (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/ProfileList (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/StopMultiNode (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/StopNode (gopogh)	0.00 (chart)
Hyperkit_macOS	TestOffline (gopogh)	0.00 (chart)
Hyperkit_macOS	TestScheduledStopUnix (gopogh)	0.00 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	1.16 (chart)
Hyperkit_macOS	TestMultiNode/serial/ValidateNameConflict (gopogh)	2.38 (chart)
Docker_Linux	TestNetworkPlugins/group/kindnet/Start (gopogh)	2.44 (chart)
Docker_Linux	TestNetworkPlugins/group/bridge/DNS (gopogh)	3.25 (chart)
Docker_Linux	TestNetworkPlugins/group/enable-default-cni/DNS (gopogh)	3.25 (chart)
Hyperkit_macOS	TestMultiNode/serial/DeleteNode (gopogh)	3.57 (chart)
Hyperkit_macOS	TestMultiNode/serial/RestartMultiNode (gopogh)	3.57 (chart)
Docker_Linux	TestKubernetesUpgrade (gopogh)	4.07 (chart)
Docker_Linux	TestNetworkPlugins/group/calico/Start (gopogh)	4.07 (chart)
Docker_Linux	TestNetworkPlugins/group/kubenet/DNS (gopogh)	4.10 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

minikube-pr-bot · 2021-12-22T23:33:13Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 45.3s    | 44.7s               |
| enable ingress | 28.7s    | 29.5s               |
+----------------+----------+---------------------+

Times for minikube start: 48.1s 45.7s 45.4s 44.2s 43.3s
Times for minikube (PR 13123) start: 44.2s 44.3s 45.5s 45.0s 44.5s

Times for minikube ingress: 28.5s 29.6s 30.6s 29.0s 25.6s
Times for minikube (PR 13123) ingress: 29.1s 31.6s 29.6s 28.6s 28.5s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 26.6s    | 26.0s               |
| enable ingress | 22.2s    | 33.8s               |
+----------------+----------+---------------------+

Times for minikube ingress: 21.4s 22.9s 22.9s 21.9s 21.9s
Times for minikube (PR 13123) ingress: 19.9s 22.9s 82.4s 21.9s 21.9s

Times for minikube start: 29.9s 25.8s 25.8s 26.2s 25.3s
Times for minikube (PR 13123) start: 26.0s 25.6s 25.5s 26.3s 26.7s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13123) |
+----------------+----------+---------------------+
| minikube start | 41.6s    | 42.1s               |
| enable ingress | 22.9s    | 24.3s               |
+----------------+----------+---------------------+

Times for minikube start: 34.7s 41.4s 45.4s 45.7s 40.9s
Times for minikube (PR 13123) start: 42.0s 40.5s 45.2s 42.0s 40.6s

Times for minikube ingress: 19.9s 33.4s 18.9s 18.9s 23.4s
Times for minikube (PR 13123) ingress: 22.9s 23.4s 18.9s 22.9s 33.4s

k8s-ci-robot · 2021-12-23T00:45:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: klaases, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2021-12-23T01:43:12Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyperkit_macOS	TestAddons/Setup (gopogh)	0.00 (chart)
Hyperkit_macOS	TestErrorSpam/setup (gopogh)	0.00 (chart)
Hyperkit_macOS	TestFunctional/serial/AuditLog (gopogh)	0.00 (chart)
Hyperkit_macOS	TestFunctional/serial/StartWithProxy (gopogh)	0.00 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/pause/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/pause/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/parallel/DistinctCurrentSteps (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/start/parallel/IncreasingCurrentSteps (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/unpause/Audit (gopogh)	0.00 (chart)
Hyperkit_macOS	TestJSONOutput/unpause/Command (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMountStart/serial/StartWithMountFirst (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/CopyFile (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/ProfileList (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/StopMultiNode (gopogh)	0.00 (chart)
Hyperkit_macOS	TestMultiNode/serial/StopNode (gopogh)	0.00 (chart)
Hyperkit_macOS	TestOffline (gopogh)	0.00 (chart)
Hyperkit_macOS	TestScheduledStopUnix (gopogh)	0.00 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/custom-weave/NetCatPod (gopogh)	0.00 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/calico/NetCatPod (gopogh)	1.16 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	1.30 (chart)
Hyperkit_macOS	TestMultiNode/serial/DeleteNode (gopogh)	1.33 (chart)
Hyperkit_macOS	TestMultiNode/serial/RestartMultiNode (gopogh)	1.33 (chart)
Hyperkit_macOS	TestMultiNode/serial/ValidateNameConflict (gopogh)	1.33 (chart)
Docker_Linux_containerd	TestScheduledStopUnix (gopogh)	1.80 (chart)
Docker_Linux_containerd	TestStartStop/group/newest-cni/serial/VerifyKubernetesImages (gopogh)	2.41 (chart)
Hyperkit_macOS	TestMultiNode/serial/RestartKeepsNodes (gopogh)	2.63 (chart)
KVM_Linux	TestPause/serial/Start (gopogh)	2.68 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

klaases added 2 commits November 11, 2021 14:34

update messaging per runtime

906b57e

Bootloader ISO EFI Support

694c9cc

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 7, 2021

k8s-ci-robot assigned klaases Dec 7, 2021

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 7, 2021

k8s-ci-robot requested review from blueelvis and spowelljr December 7, 2021 23:58

medyagh changed the title ~~Bootloader ISO EFI Support~~ wip: Bootloader ISO EFI Support Dec 8, 2021

klaases changed the title ~~wip: Bootloader ISO EFI Support~~ WIP: Bootloader ISO EFI Support Dec 8, 2021

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2021

klaases added 11 commits December 9, 2021 15:07

Add "post-image-efi.sh" script.

339afcf

Merge branch 'master' of github.com:kubernetes/minikube into boot1

dcdef43

Remove comments for needed files.

70e8b1f

Merge branch 'master' of github.com:klaases/minikube into boot1

404b9c9

Remove resolved question.

4e7e636

Add grub-efi.cfg

786815e

Add genimage-efi.cfg

36957e9

Add grub-bios.cfg

b67bd79

Revert unintended change.

09ba078

Remove duplicate entry.

2789557

Add boilerplate for new sh files (solves for lint).

ff24188

klaases added 2 commits December 10, 2021 12:03

Add defconfig for testing.

e4f76a7

Merge branch 'master' of github.com:kubernetes/minikube into boot1

9eaf954

klaases mentioned this pull request Dec 21, 2021

Build ISO on Debian 10 agent with Docker #13218

Closed

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 22, 2021

Fix Makefile ISO_VERSION and isoBucket references.

036b537

sharifelgamal approved these changes Dec 23, 2021

View reviewed changes

sharifelgamal merged commit cab934e into kubernetes:master Dec 23, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 23, 2021

klaases mentioned this pull request Jan 5, 2022

REQUEST: New membership for klaases kubernetes/org#3162

Closed

7 tasks

spowelljr mentioned this pull request Jan 20, 2022

Revert bootloader ISO EFI Support PR #13414

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootloader ISO EFI Support #13123

Bootloader ISO EFI Support #13123

klaases commented Dec 7, 2021 •

edited

k8s-ci-robot commented Dec 7, 2021

klaases commented Dec 7, 2021

klaases commented Dec 7, 2021 •

edited

minikube-bot commented Dec 8, 2021

klaases commented Dec 8, 2021

klaases commented Dec 10, 2021

minikube-bot commented Dec 21, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

klaases commented Dec 22, 2021

klaases commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

klaases commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

klaases commented Dec 22, 2021 •

edited

minikube-pr-bot commented Dec 22, 2021

afbjorklund commented Dec 22, 2021

minikube-pr-bot commented Dec 22, 2021

minikube-pr-bot commented Dec 22, 2021

k8s-ci-robot commented Dec 23, 2021

minikube-pr-bot commented Dec 23, 2021

Bootloader ISO EFI Support #13123

Bootloader ISO EFI Support #13123

Conversation

klaases commented Dec 7, 2021 • edited

k8s-ci-robot commented Dec 7, 2021

klaases commented Dec 7, 2021

klaases commented Dec 7, 2021 • edited

minikube-bot commented Dec 8, 2021

klaases commented Dec 8, 2021

klaases commented Dec 10, 2021

minikube-bot commented Dec 21, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

klaases commented Dec 22, 2021

klaases commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

klaases commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

minikube-bot commented Dec 22, 2021

sharifelgamal commented Dec 22, 2021

klaases commented Dec 22, 2021 • edited

minikube-pr-bot commented Dec 22, 2021

afbjorklund commented Dec 22, 2021

minikube-pr-bot commented Dec 22, 2021

minikube-pr-bot commented Dec 22, 2021

k8s-ci-robot commented Dec 23, 2021

minikube-pr-bot commented Dec 23, 2021

klaases commented Dec 7, 2021 •

edited

klaases commented Dec 7, 2021 •

edited

klaases commented Dec 22, 2021 •

edited