Experiment with managing external machines with lima #2000

afbjorklund · 2023-11-15T22:09:48Z

Similar to the docker-machine "generic" driver, bring your own virtual machine (or physical server)

Not so useful in itself, but not so bad when made into a real driver or wrapped with helper scripts...

NAME          STATUS     SSH                   VMTYPE    ARCH      CPUS    MEMORY    DISK      DIR
beaglebone    Running    192.168.7.2:22        ext       armv7l    1       512MiB    4GiB      ~/.lima/beaglebone
core          Stopped    127.0.0.1:0           qemu      x86_64    1       1GiB      100GiB    ~/.lima/core

vmType: ext

arch: "armv7l"
cpus: 1
memory: 512MiB
disk: 4GiB

# We do not have arm-v7 binaries of containerd
containerd:
  system: false
  user: false

ssh:
  address: 192.168.7.2

Requires:

Restore guest agent unix socket functionality #2006

Installed lima-guestagent, and nerdctl from tarballs/binaries.

$ _output/bin/limactl shell beaglebone nerdctl version
Client:
 Version:	v1.7.0
 OS/Arch:	linux/arm
 Git commit:	e674fe7ba6e49f12e88cd9c6c442e7ea5232502c
 buildctl:
  Version:	v0.12.3
  GitCommit:	438f47256f0decd64cc96084e22d3357da494c27

Server:
 containerd:
  Version:	v1.7.6
  GitCommit:	091922f03c2762540fd057fba91260237ff86acb
 runc:
  Version:	1.1.9
  GitCommit:	v1.1.9-0-gccaecfc

https://www.beagleboard.org/boards/beaglebone-black

You could also use a Raspberry Pi Zero*, or a cloud droplet.

* need the Zero 2, for arm-v7 (previous model was arm-v6)

https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/

afbjorklund · 2023-11-18T12:22:23Z

The probes are somewhat annoying when not using cidata, but that was the same story on FreeBSD and others.

Support for FreeBSD guests ? #1508 (comment)

Maybe there should be some fallback implementation, at least for ssh-ready/guestagent install/boot-done ?

sudo diff -q /run/lima-ssh-ready /mnt/lima-cidata/meta-data

install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
sudo /usr/local/bin/lima-guestagent install-systemd

sudo diff -q /run/lima-boot-done /mnt/lima-cidata/meta-data

Like copying the lima-guestagent over the ssh connection (my workaround)

And keeping the instance meta-data (id) somewhere else, like in /etc ?

afbjorklund · 2023-11-22T19:38:33Z

Added support for hostnames, so that you can use nice features like avahi-daemon

NAME           STATUS     SSH                     VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
raspberrypi    Running    raspberrypi.local:22    ext       aarch64    4       512MiB    32GiB     ~/.lima/raspberrypi

anders@raspberrypi:~ $ sudo mkdir /mnt/lima-cidata
anders@raspberrypi:~ $ sudo touch /mnt/lima-cidata/meta-data
anders@raspberrypi:~ $ sudo touch /run/lima-ssh-ready
anders@raspberrypi:~ $ sudo touch /run/lima-boot-done

Previously I was assuming IP. Now, to have a nice way to install the lima-guestagent

afbjorklund · 2023-11-29T10:52:47Z

Added host key checking:

The authenticity of host 'raspberrypi.local (192.168.0.113)' can't be established.
ECDSA key fingerprint is SHA256:tKIRfOeWP1HeCFLpM0UT30CUWSDXpC7gxPsKHUnS+h4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

And shortened the timeout, for when the device is not connected. Resolves in 10ms, when cached (or ~250ms).

errors="[field `ssh.address` must be IP: lookup raspberrypi.local: i/o timeout]"

afbjorklund · 2024-01-07T21:40:20Z

Added provision scripts.

Using sudo for system.

AkihiroSuda · 2024-01-22T14:17:19Z

pkg/sshutil/sshutil.go

@@ -231,7 +237,7 @@ func SSHOpts(instDir string, useDotSSH, forwardAgent, forwardX11, forwardX11Trus
 	if err != nil {
 		return nil, err
 	}
-	opts, err := CommonOpts(useDotSSH)
+	opts, err := CommonOpts(useDotSSH, hostAddress == "127.0.0.1")


What about ::1?

It was only meant to handle the default string, not any overrides like "localhost" or "::1" (ipv6)

There is probably a utility net function that one can call, instead of hardcoding strings like this.

AkihiroSuda · 2024-01-22T14:17:44Z

pkg/limayaml/limayaml.go

@@ -66,6 +66,7 @@ const (
 	QEMU VMType = "qemu"
 	VZ   VMType = "vz"
 	WSL2 VMType = "wsl2"
+	EXT  VMType = "ext"


Maybe this should be called "external" to avoid confusion with "extended"

In minikube we ended up calling the driver "ssh", but I don't think it's great

AkihiroSuda · 2024-01-22T14:19:38Z

examples/default.yaml

@@ -117,6 +117,9 @@ additionalDisks:
 #   fsType: "ext4"

 ssh:
+  # Address for the host.
+  # 🟢 Builtin default: "127.0.0.1" (localhost)
+  address: null


What does localPort mean now for non-local address?

It used as "port", but the old name is kept for compatibility.

If the port is zero, then for non-localhost it will be set to 22.
For localhost (127.0.0.1), it is still assigned to a random port.

Normally it is not set, and only the IP address is used for SSH.

afbjorklund · 2024-03-18T18:20:34Z

Rebased to lima v0.21.0 [RPi runs Debian GNU/Linux 12 (bookworm)]

$ _output/bin/limactl shell raspberrypi nerdctl version
Client:
 Version:	v1.7.5
 OS/Arch:	linux/arm64
 Git commit:	cffed372371dcbea3dc9a646ce5a913fc1c09513
 buildctl:
  Version:	v0.12.5
  GitCommit:	bac3f2b673f3f9d33e79046008e7a38e856b3dc6

Server:
 containerd:
  Version:	v1.7.14
  GitCommit:	dcf2847247e18caba8dce86522029642f60fe96b
 runc:
  Version:	1.1.12
  GitCommit:	v1.1.12-0-g51d5e946

Quite svelte, without the cidata etc: 44K /home/anders/.lima/raspberrypi

NAME           STATUS     SSH                     VMTYPE    ARCH       CPUS    MEMORY    DISK     DIR
raspberrypi    Running    raspberrypi.local:22    ext       aarch64    4       512MiB    32GiB    ~/.lima/raspberrypi

AkihiroSuda · 2024-04-22T00:40:59Z

Would it be possible to test this on CI?

afbjorklund · 2024-04-22T05:56:48Z

Would it be possible to test this on CI?

As long as it is possible to supply a VM, with access through host keys and authorized keys, that should be possible.

I should detail the required steps (with example log), especially now with the addition of the cloud-config generation.

Generate cloud-config outside of cidata.iso too #2271

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Verify ssh host keys, when connecting to a remote server. The first connection will prompt, if not in known_hosts. Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

It is not using cloud-init anyway, and does not need another copy of lima-guestagent and nerdctl-full.tgz Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

afbjorklund force-pushed the external branch from 64ec4ff to b865895 Compare November 15, 2023 22:11

afbjorklund mentioned this pull request Nov 17, 2023

Support for native driver specific guest agent connection #1998

Merged

4 tasks

afbjorklund force-pushed the external branch 2 times, most recently from ea5255d to 9cea961 Compare November 24, 2023 07:33

afbjorklund mentioned this pull request Nov 24, 2023

Add hidden command to install the guest components #2028

Draft

afbjorklund force-pushed the external branch from 9cea961 to dcdb6f8 Compare November 24, 2023 07:41

afbjorklund mentioned this pull request Nov 25, 2023

Libvirt driver, for hypervisors on remote machines #2031

Open

afbjorklund force-pushed the external branch from dcdb6f8 to 5b84678 Compare November 29, 2023 10:51

afbjorklund force-pushed the external branch 2 times, most recently from 0ab8193 to b91338b Compare January 2, 2024 07:32

afbjorklund mentioned this pull request Jan 2, 2024

External driver, for cloud and physical #2109

Open

afbjorklund force-pushed the external branch from 3998d45 to 5173ce3 Compare January 7, 2024 21:39

afbjorklund force-pushed the external branch from c2abd42 to 7b2e56d Compare January 19, 2024 16:39

afbjorklund marked this pull request as ready for review January 21, 2024 18:59

AkihiroSuda reviewed Jan 22, 2024

View reviewed changes

afbjorklund force-pushed the external branch from 7b2e56d to a18e091 Compare February 24, 2024 08:38

afbjorklund force-pushed the external branch from a18e091 to 9b8c9ea Compare March 18, 2024 18:10

AkihiroSuda added this to the v0.21.1 milestone Apr 22, 2024

afbjorklund removed this from the v0.22.0 milestone Apr 26, 2024

afbjorklund marked this pull request as draft April 26, 2024 06:01

afbjorklund force-pushed the external branch from 9b8c9ea to e11848a Compare April 28, 2024 12:41

afbjorklund added 12 commits May 7, 2024 12:23

Add SSH address for supplying external IP

3c85d90

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Use SSH address also for copy command

612c289

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Use SSH address also for host agent

eb834c1

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Use host key checking outside localhost

5e55297

Verify ssh host keys, when connecting to a remote server. The first connection will prompt, if not in known_hosts. Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Print IP instead of Port for non-local SSH

056714d

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Add ext driver for external VM machines

cad2bac

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Allow using address like raspberrypi.local

00ba571

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Add a shorter timeout for mDNS IP lookup

8128e7a

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Don't wait for bootscripts for external

31002fe

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Don't show log for already cached archive

b0b726c

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Add external support for provision scripts

f31eee1

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Don't generate cidata.iso for external vm

3927cf1

It is not using cloud-init anyway, and does not need another copy of lima-guestagent and nerdctl-full.tgz Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

afbjorklund force-pushed the external branch from e11848a to 3927cf1 Compare May 7, 2024 10:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experiment with managing external machines with lima #2000

Experiment with managing external machines with lima #2000

afbjorklund commented Nov 15, 2023 •

edited

afbjorklund commented Nov 18, 2023 •

edited

afbjorklund commented Nov 22, 2023 •

edited

afbjorklund commented Nov 29, 2023 •

edited

afbjorklund commented Jan 7, 2024

AkihiroSuda Jan 22, 2024

afbjorklund Jan 23, 2024

AkihiroSuda Jan 22, 2024

afbjorklund Feb 11, 2024

AkihiroSuda Jan 22, 2024

afbjorklund Jan 23, 2024

afbjorklund commented Mar 18, 2024 •

edited

AkihiroSuda commented Apr 22, 2024

afbjorklund commented Apr 22, 2024 •

edited

Experiment with managing external machines with lima #2000

Are you sure you want to change the base?

Experiment with managing external machines with lima #2000

Conversation

afbjorklund commented Nov 15, 2023 • edited

afbjorklund commented Nov 18, 2023 • edited

afbjorklund commented Nov 22, 2023 • edited

afbjorklund commented Nov 29, 2023 • edited

afbjorklund commented Jan 7, 2024

AkihiroSuda Jan 22, 2024

Choose a reason for hiding this comment

afbjorklund Jan 23, 2024

Choose a reason for hiding this comment

AkihiroSuda Jan 22, 2024

Choose a reason for hiding this comment

afbjorklund Feb 11, 2024

Choose a reason for hiding this comment

AkihiroSuda Jan 22, 2024

Choose a reason for hiding this comment

afbjorklund Jan 23, 2024

Choose a reason for hiding this comment

afbjorklund commented Mar 18, 2024 • edited

AkihiroSuda commented Apr 22, 2024

afbjorklund commented Apr 22, 2024 • edited

afbjorklund commented Nov 15, 2023 •

edited

afbjorklund commented Nov 18, 2023 •

edited

afbjorklund commented Nov 22, 2023 •

edited

afbjorklund commented Nov 29, 2023 •

edited

afbjorklund commented Mar 18, 2024 •

edited

afbjorklund commented Apr 22, 2024 •

edited