Multipass instance can not be started while F5 VPN is on

[CharlieYu12345]

Describe the bug
When I install the microk8s with Multipass, if the F5 VPN is off, the installation of microk8s-vm can be successful. but if the F5 VPN is on, the installation is failed.

Warning: the "--mem" long option is deprecated in favour of "--memory". Please update any scripts, etc.
launch failed: The following errors occurred:
An error occurred with the instance when trying to launch with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.

To Reproduce
How, and what happened?

1. multipass microk8s-vm
2. BIG-IP F5 VPN is on

Expected behavior
the microk8s-vm can be started successfully with F5 VPN on because the Mac VPN is managed by the company.

Logs
`[2024-05-06T16:47:28.971] [debug] [daemon] Returning setting local.driver=qemu
[2024-05-06T16:48:05.736] [debug] [qemu-system-aarch64] [32408] started: qemu-system-aarch64 --version
[2024-05-06T16:48:06.075] [debug] [qemu-img] [32412] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20240426/ubuntu-22.04-server-cloudimg-arm64.img
[2024-05-06T16:48:06.098] [debug] [qemu-img] [32413] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/ubuntu-22.04-server-cloudimg-arm64.img 53687091200
[2024-05-06T16:48:06.120] [debug] [qemu-img] [32415] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/ubuntu-22.04-server-cloudimg-arm64.img
[2024-05-06T16:48:06.131] [debug] [qemu-img] [32416] started: qemu-img amend -o compat=1.1 /var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/ubuntu-22.04-server-cloudimg-arm64.img
[2024-05-06T16:48:06.139] [debug] [microk8s-vm] process working dir ''
[2024-05-06T16:48:06.139] [info] [microk8s-vm] process program 'qemu-system-aarch64'
[2024-05-06T16:48:06.139] [info] [microk8s-vm] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:61:9a:6c, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 2, -m, 4096M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/cloud-init-config.iso'
[2024-05-06T16:48:06.144] [debug] [qemu-system-aarch64] [32417] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.hoHNbC
[2024-05-06T16:48:06.188] [info] [microk8s-vm] process state changed to Starting
[2024-05-06T16:48:06.192] [info] [microk8s-vm] process state changed to Running
[2024-05-06T16:48:06.192] [debug] [qemu-system-aarch64] [32418] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:61:9a:6c -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 2 -m 4096M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/microk8s-vm/cloud-init-config.iso
[2024-05-06T16:48:06.192] [info] [microk8s-vm] process started
[2024-05-06T16:48:06.193] [debug] [microk8s-vm] Waiting for SSH to be up
[2024-05-06T16:48:06.379] [debug] [microk8s-vm] QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2024-05-06T16:48:06.416] [debug] [microk8s-vm] QMP: {"return": {}}

[2024-05-06T16:48:20.291] [debug] [microk8s-vm] QMP: {"timestamp": {"seconds": 1714985300, "microseconds": 291195}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[7]/virtio-backend"}}

[2024-05-06T16:53:15.466] [debug] [blueprint provider] Loading "anbox-cloud-appliance" v1
[2024-05-06T16:53:15.469] [debug] [blueprint provider] Loading "charm-dev" v1
[2024-05-06T16:53:15.469] [debug] [blueprint provider] Loading "docker" v1
[2024-05-06T16:53:15.470] [debug] [blueprint provider] Loading "jellyfin" v1
[2024-05-06T16:53:15.470] [debug] [blueprint provider] Loading "minikube" v1
[2024-05-06T16:53:15.472] [debug] [blueprint provider] Loading "ros-noetic" v1
[2024-05-06T16:53:15.472] [debug] [blueprint provider] Loading "ros2-humble" v1
`

Additional info

• OS: [Apple M3 Pro, MacOS 14.4.1 (23E224)]
• multipass 1.13.1+mac
• multipass info
• multipass get local.driver: qemu

Additional context
Add any other context about the problem here.

[sharder996]

Hi @CharlieYu12345

VPN's are known to play havoc with Multipass' networking. Because we can't account for every kind of VPN configuration and we don't believe we should be changing your configuration for you, we don't directly support the use of Multipass with VPNs.

However, we do have some documentation on the topic here and here, and I would recommend taking a look at that first.

If that still doesn't work for you, let us know!

[CharlieYu12345]

Thanks @sharder996,
I have tried the

Not sure if this is a workaround or solution, but I was able to make this work by nat-ing the multipass bridge with my vpn interface.

I added nat on utun1 from bridge100:network to any -> (utun1) to file /etc/pf.conf. Then I reloaded the file: $ sudo pfctl -f /etc/pf.conf.

(Just built my first snap on mac 😃)

It does not work, because my mac has no nat command.

[sharder996]

nat is not a command that you execute in your terminal. You need to edit the file /etc/pf.conf by adding the line

nat on utun1 from bridge100:network to any -> (utun1)

and then reload the file with the command sudo pfctl -f /etc/pf.conf

It is explained in full here:
https://multipass.run/docs/troubleshoot-networking#heading--potential-workaround-for-vpn-conflicts

[CharlieYu12345]

I tried the following actions, still not working:

1. Disable VPN
2. Use OnyX to clean all cache.
3. Reinstall multipass.
4. Restart OS
5. add ant line into /etc/pf.conf succucefully

Thanks.

[CharlieYu12345]

I Enabled the trace of multipass log and got more info:

[2024-05-12T09:14:53.359] [debug] [qemu-system-aarch64] [4826] started: qemu-system-aarch64 --version

[2024-05-12T09:14:53.393] [debug] [qemu-img] [4827] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/noble-20240423/ubuntu-24.04-server-cloudimg-arm64.img
[2024-05-12T09:14:53.411] [debug] [qemu-img] [4828] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-24.04-server-cloudimg-arm64.img 5368709120
[2024-05-12T09:14:53.423] [debug] [qemu-img] [4829] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-24.04-server-cloudimg-arm64.img
[2024-05-12T09:14:53.430] [debug] [qemu-img] [4830] started: qemu-img amend -o compat=1.1 /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-24.04-server-cloudimg-arm64.img
[2024-05-12T09:14:53.437] [debug] [primary] process working dir ''
[2024-05-12T09:14:53.437] [info] [primary] process program 'qemu-system-aarch64'
[2024-05-12T09:14:53.437] [info] [primary] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:40:5e:98, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-24.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 1, -m, 1024M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso'
[2024-05-12T09:14:53.439] [debug] [qemu-system-aarch64] [4831] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.WaaaUy
[2024-05-12T09:14:53.469] [info] [primary] process state changed to Starting
[2024-05-12T09:14:53.471] [info] [primary] process state changed to Running
[2024-05-12T09:14:53.471] [debug] [qemu-system-aarch64] [4832] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:40:5e:98 -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-24.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso
[2024-05-12T09:14:53.471] [info] [primary] process started
[2024-05-12T09:14:53.472] [debug] [primary] Waiting for SSH to be up
[2024-05-12T09:14:53.594] [debug] [primary] QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2024-05-12T09:14:53.617] [debug] [primary] QMP: {"return": {}}

[2024-05-12T09:14:54.546] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:14:56.674] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:14:58.728] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:00.855] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:02.974] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:03.676] [debug] [primary] QMP: {"timestamp": {"seconds": 1715476503, "microseconds": 676875}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[6]/virtio-backend"}}

[2024-05-12T09:15:05.116] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:07.168] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:09.267] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:11.345] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:13.454] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:15.517] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:17.623] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:19.713] [trace] [primary] Could not determine IP address within 1000ms
[2024-05-12T09:15:21.782] [trace] [primary] Could not determine IP address within 1000ms

[sharder996]

Hi @CharlieYu12345, can you confirm whether or not you can launch and connect to instances when your VPN is disabled?