re-running sudo ./vhive issue

[facelessq]

After I use ctrl+c to terminate sudo ./vhive, and I want to re-run sudo ./vhive,
I will first get failed to listen error like this

INFO[2021-04-07T07:44:35.476766276-06:00] Registering bridges for tap manager          
INFO[2021-04-07T07:44:35.479102046-06:00] Creating containerd client                   
INFO[2021-04-07T07:44:35.480040706-06:00] Created containerd client                    
INFO[2021-04-07T07:44:35.480084418-06:00] Creating firecracker client                  
INFO[2021-04-07T07:44:35.480227389-06:00] Created firecracker client                   
INFO[2021-04-07T07:44:35.480537144-06:00] Listening on port:3334                       
FATA[2021-04-07T07:44:35.480544935-06:00] failed to listen: listen unix /etc/firecracker-containerd/fccd-cri.sock: bind: address already in use

And if I run sudo ./vhive again and again, I will then always get Bridge could not be created error.

INFO[2021-04-07T07:44:50.113721845-06:00] Registering bridges for tap manager          
PANI[2021-04-07T07:44:50.113909020-06:00] Bridge could not be created                   bridge=br0
panic: (*logrus.Entry) 0xc0003a1e30

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003a1dc0, 0x0, 0xc0002e4d40, 0x1b)
	/users/faceless/go/pkg/mod/github.com/sirupsen/logrus@v1.8.0/entry.go:259 +0x2e5
github.com/sirupsen/logrus.(*Entry).Log(0xc0003a1dc0, 0x0, 0xc0002df958, 0x1, 0x1)
	/users/faceless/go/pkg/mod/github.com/sirupsen/logrus@v1.8.0/entry.go:285 +0x86
github.com/sirupsen/logrus.(*Entry).Panic(...)
	/users/faceless/go/pkg/mod/github.com/sirupsen/logrus@v1.8.0/entry.go:323
github.com/ease-lab/vhive/taps.createBridge(0xc0003a8f49, 0x3, 0xc0003a8f50, 0xb)
	/users/faceless/vhive/taps/tapManager.go:88 +0x668
github.com/ease-lab/vhive/taps.NewTapManager(0x12289a0)
	/users/faceless/vhive/taps/tapManager.go:70 +0x212
github.com/ease-lab/vhive/misc.NewVMPool(...)
	/users/faceless/vhive/misc/vm_pool.go:34
github.com/ease-lab/vhive/ctriface.NewOrchestrator(0x12ee6b7, 0x9, 0x0, 0x0, 0xc0002dff40, 0x5, 0x5, 0xc00038ca90)
	/users/faceless/vhive/ctriface/orch.go:78 +0x68
main.main()
	/users/faceless/vhive/vhive.go:122 +0x716

Why does this happen?

[sosson97]

I've experienced the exactly same issue. At the first execution right after reboot, running ./vhive aborts with "failed to listen" error. It aborts with "Bridge could not be created" from next executions.

[ustiugov]

@sosson97 please see my response below. Also, we would like to keep in touch with those who are trying out vHive to facilitate its adoption. We would be happy if you join our channel firecracker-vhive-research in the Firecracker Slack and let me know which university or organization you are at (e.g., in that Slack channel or in DM).

[sosson97]

Thanks @ustiugov for reply. I'd like to join the slack channel, but it seems that the link in your reply is not working.

[ustiugov]

@sosson97 sorry, here is the link

[sosson97]

It's still not working, and neither is the link in firecracker's github page.

[ustiugov]

@sosson97 something is messed up on the firecracker team's end. Here is the link what I found in their recent commit that is supposed to fix it.

[ustiugov]

@facelessq @sosson97 thank you for the questions.

Upon start, vHive needs to connect to the firecracker-containerd daemon to control the lifecycle of VMs. If vHive was terminated abnormally, some host resources might not be cleaned up. Particularly, the unix socket file as in the first log and the bridges as in the second example. Bridges are normally created upon a vHive start so that VMs's taps can be connected to the host interface via a bridge. Note that taps and bridges are created by vHive directly and not by the CNI plugin.

Solution:

1. Clean up: ./scripts/github_runner/clean_cri_runner.sh
2. Restart the cluster, following the instructions for a single-node cluster. We don't have a set of instructions for restarting a multi-node cluster as of now so, I would recommend starting from scratch in that case.

Another recommendation is to use kind containers instead of a bare-metal host for development, see the corresponding instructions.