Allow to passing an MTU value to be used by the buildkit Docker image (Gitlab CI/DinD)

[zedtux]

What is happening

I'm using a Kubernetes cluster, configured in Gitlab, in order to build/test/deploy apps.

I've build my .gitlab-ci.yml file so that it uses the earthly/earthly:v... docker image and run earthly commands to do the actions.

My build is stuck on downloading Alpine package definition while building the Docker image of my project. I discovered that an issue with Docker in Docker (or DinD) could come from different values of the network interface MTU (Maximum Transmission Unit).

I tried a wget -S https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz in the Pod where is running the earthly/earthly:v0.5.4 Docker image and it works fine.
Then I tried a docker exec -it earthly-buildkitd sh and tried the same command but it remain stuck while waiting for the download:

/ # hostname
runner-uiq6dk-project-24146750-concurrent-1f4rrs
/ # docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS             PORTS                      NAMES
e715b97a6a0b   earthly/buildkitd:v0.5.4   "/usr/bin/entrypoint…"   55 minutes ago   Up About an hour   127.0.0.1:8373->8373/tcp   earthly-buildkitd
/ # docker exec -it earthly-buildkitd sh -c 'wget -S http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz ; rm -f APKINDEX.tar.gz'
Connecting to dl-cdn.alpinelinux.org (151.101.130.133:80)
  HTTP/1.1 200 OK
  Server: nginx
  Content-Type: application/octet-stream
  Last-Modified: Wed, 03 Mar 2021 04:41:11 GMT
  ETag: "603f1367-9ade4"
  Strict-Transport-Security: max-age=31536000
  X-Frame-Options: DENY
  X-Content-Type-Options: nosniff
  Via: 1.1 varnish, 1.1 varnish
  Content-Length: 634340
  Accept-Ranges: bytes
  Date: Fri, 05 Mar 2021 06:51:00 GMT
  Age: 0
  Connection: close
  X-Served-By: cache-lga21930-LGA, cache-fra19158-FRA
  X-Cache: HIT, HIT
  X-Cache-Hits: 1, 1
  X-Timer: S1614927060.211487,VS0,VE157
wget: can't open 'APKINDEX.tar.gz': File exists
/ # docker exec -it earthly-buildkitd sh -c 'wget -S http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz ; rm -f APKINDEX.tar.gz'
Connecting to dl-cdn.alpinelinux.org (151.101.130.133:80)
  HTTP/1.1 200 OK
  Server: nginx
  Content-Type: application/octet-stream
  Last-Modified: Wed, 03 Mar 2021 04:41:11 GMT
  ETag: "603f1367-9ade4"
  Strict-Transport-Security: max-age=31536000
  X-Frame-Options: DENY
  X-Content-Type-Options: nosniff
  Via: 1.1 varnish, 1.1 varnish
  Content-Length: 634340
  Accept-Ranges: bytes
  Date: Fri, 05 Mar 2021 06:51:02 GMT
  Age: 2
  Connection: close
  X-Served-By: cache-lga21930-LGA, cache-fra19130-FRA
  X-Cache: HIT, HIT
  X-Cache-Hits: 1, 1
  X-Timer: S1614927062.267567,VS0,VE0
saving to 'APKINDEX.tar.gz'
APKINDEX.tar.gz        0% |                                                                                                                                                                                          |     0  - stalled -

Trying a manuel fix

Checking the MTU values using ifconfig revealed the case of different MTU values:

/ # docker exec -it earthly-buildkitd sh -c 'ifconfig'
cni0      Link encap:Ethernet  HWaddr DE:00:11:7A:1B:E9  
          inet addr:172.30.0.1  Bcast:172.30.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16327 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26178 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1437257 (1.3 MiB)  TX bytes:173257992 (165.2 MiB)

eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02  
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:29517 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17710 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:200845517 (191.5 MiB)  TX bytes:1802752 (1.7 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth1f328984 Link encap:Ethernet  HWaddr 4A:C3:66:35:46:C4  
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7248 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9399 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:814188 (795.1 KiB)  TX bytes:53328607 (50.8 MiB)

So the cni0 network interface has a MTU: 1500 and the eth0 network interface has a MTU:1450. Changing the cni0 MTU to 1450 (docker exec -it earthly-buildkitd sh -c 'ifconfig cni0 mtu 1450') made the same wget command working.

Sinking into the Earthly bowels

After having quickly review the way the earthly/buildkitd Docker image is built, I found that buildkitd is configured to use the /etc/cni/cni-conf.json file.

I've cloned this repo, added an hardcoded MTU line to this file:

{
	"cniVersion": "0.3.0",
	"name": "buildkitbuild",
	"type": "bridge",
	"bridge": "cni0",
	"isGateway": true,
	"ipMasq": true,
        "mtu": 1450,
	"ipam": {
		"type": "host-local",
		"subnet": "172.30.0.0/16",
		"routes": [
			{ "dst": "0.0.0.0/0" }
		]
	}
}

and rebuilt the image that I've pushed to docker.io.
In my .gitlab-ci.yml, I've defined the EARTHLY_BUILDKIT_IMAGE env variable in order to use my Docker image, and now my build is working fine.

Proposal

One could pass a desired MTU for the buildkitd by specifying an --mtu or --buildkits-mtu flag which would update the /etc/cni/cni-conf.json file accordingly.

[dchw]

I wonder if auto-detecting host MTU is possible here and just matching it?

[zedtux]

Even better !