ssh handshake failure when running "flux bootstrap git"

[smveloso]

Describe the bug

Upon running "flux bootstrap git" with a "--url" parameter pointing to an existing github repository, I get the error message:

✗ failed to clone repository: ssh: handshake failed: knownhosts: key mismatch

The github repository exists, is private and is accessible via "git clone ...".

The host key exists in "~/.ssh/known_hosts":

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

There are no flux components running in the kubernetes cluster, this is the actual bootstraping. I have opted to use "bootstrap git" instead of "bootstrap github" mostly out of curiority.

If the host key is not present in "~/.ssh/known_hosts" I get a different (and quite expected) error message:

✗ failed to clone repository: ssh: handshake failed: knownhosts: key is unknown

It seems clear to me that whatever the problem is it's happening in the cli itself, nothing to do with the k8s cluster.

Steps to reproduce

1. Create a repository on github (e.g. acme)
2. Create or get access to a k8s cluster (kubeconfig at ~/.kube/acme)
3. Make sure github.com hostkey exists in "~/.ssh/known_hosts"
4. Run: flux bootstrap git --kubeconfig ~/.kube/acme --url=ssh://git@github.com/yourname/acme

The hostkey:

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

Expected behavior

Correct bootstraping.

Screenshots and recordings

No response

OS / Distro

ArchLinux

Flux version

0.28.5

Flux check

flux --kubeconfig ~/.kube/flux-lab-one check ~/Env/fluxv2
► checking prerequisites
✔ Kubernetes 1.22.7+k3s1 >=1.20.6-0
► checking controllers
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[smveloso]

Refs I will probably need:

• https://github.blog/2021-09-01-improving-git-protocol-security-github/#libgit2-and-other-git-clients

• GitHub SSH Handshake failed: knownhosts key mismatch fluxcd/flux2#2097

• failed to clone repository: ssh: handshake failed: knownhosts: key mismatch fluxcd/flux2#2114 (might not be ed25519 related)

• git-ssh: try multiple algorithms for known_hosts go-git/go-git#411

• https://github.com/go-git/go-git

• https://github.com/spf13/cobra

[smveloso]

Code:

https://github.com/smveloso/flux2/blob/smveloso/cmd/flux/bootstrap_git.go

then to:

https://github.com/smveloso/flux2/blob/smveloso/internal/bootstrap/git/gogit/gogit.go

[smveloso]

Using a self-hosted gitea instead of github, I can reproduce the error when the host key is of type ed25519.

git-infra.trt3.jus.br ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID/9BRYvWhc/zupJgEz09GtmneUqSAoFUojPOblDFB1F

[smveloso]

Problem appears to be (according to reports) on the ssh client in golang.

[smveloso]

Using this entry in "known_hosts" cloning worked:

github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=

Suggested here: fluxcd#2097 (comment)

[smveloso]

Indeed, the problem seems to happen only when the key type is ed25519.

But it is supported by github:

Adding [ECDSA](https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm) and [Ed25519](https://en.wikipedia.org/wiki/EdDSA#Ed25519) host keys for SSH

[smveloso]

According to go-git/go-git#411 (comment) the problem may be related to precedence. The most preferred key type not being listed in the known_hosts file, the ssh client maybe refuses the lower precedence one that is listed.

[smveloso]

Problem is indeed located inside a go library:

File: .../go/pkg/mod/golang.org/x/crypto@v0.0.0-20210421170649-83a5a9bb288b/ssh/knownhosts/knownhosts.go

	// If the remote host starts using a different, unknown key type, we
	// also interpret that as a mismatch.
	if known, ok := knownKeys[remoteKey.Type()]; !ok || !keyEq(known.Key, remoteKey) {
		return keyErr
	}

This is very relevant: golang/go#29286
and this is interesting: is this fixed by setting ClientConfig.HostKeyAlgorithms = []string{KeyAlgoED25519}?