git storage fail due to knownhosts: key mismatch

[meetme2meat]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Following is my configuration file

---
server:
  httpListenAddr: ":3598"


storage:
  driver: "git"
  git:
    protocol: ssh
    url: github.com:xxxx/access-policies.git // our private repo
    branch: main
    subDir: policies
    checkoutDir: /Users/admin/Documents/execs/cerbos/tmp
    updatePollInterval: 60s
    ssh:
      user: git
      privateKeyFile: /Users/admin/Documents/execs/cerbos/keys/id_rsa // deploy private key 

./cerbos server --config=config.yaml

2021-12-21T09:56:24.182+0530	INFO	cerbos.server	maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2021-12-21T09:56:24.220+0530	INFO	cerbos.git.store	Cloning git repo from github.com:xxxx/access-policies.git	{"dir": "/Users/admin/Documents/execs/cerbos/tmp"}
2021-12-21T09:56:25.005+0530	ERROR	cerbos.git.store	Failed to initialize git store	{"dir": "/Users/admin/Documents/execs/cerbos/tmp", "error": "failed to clone from github.com:xxxx/access-policies.git to /Users/admin/Documents/execs/cerbos/tmp: ssh: handshake failed: knownhosts: key mismatch"}
2021-12-21T09:56:25.005+0530	INFO	cerbos.server	maxprocs: No GOMAXPROCS change to reset
ERROR: failed to create store: failed to clone from github.com:xxxx/access-policies.git to /Users/admin/Documents/execs/cerbos/tmp: ssh: handshake failed: knownhosts: key mismatch

Is there an option to set ignore hostkey check?.

I tried cloning the repo using the above private key and it work. No issue there.

Expected Behavior

It should clone the repo

Steps To Reproduce

Set the config.yaml to use git storage

---
server:
  httpListenAddr: ":3598"


storage:
  driver: "git"
  git:
    protocol: ssh
    url: github.com:xxxx/access-policies.git
    branch: main
    subDir: policies
    checkoutDir: /Users/admin/Documents/execs/cerbos/tmp
    updatePollInterval: 60s
    ssh:
      user: git
      privateKeyFile: /Users/admin/Documents/execs/cerbos/keys/id_rsa

Start cerbos using the above config

./cerbos server --config=config.yaml

Environment

- OS: MacOS
- Cerbos version: cerbos version 0.10.0
Built on 2021-11-16T12:52:36Z from b24701bb75b135518117f7d56a2d8680f9a59450

Anything else?

No response

[meetme2meat]

I guess cerbos should allow setting HostKeyCallbackHelper go-git/go-git#431 (comment)

[charithe]

Thanks for reporting the issue and digging into it.

Even though introducing the ability to turn off host key verification is the easy fix here, I want to explore the options and understand the problem a bit more to see if we can avoid introducing an insecure setting.

[charithe]

I could swear that cloning via SSH used to work before because I personally tested it myself. I think this issue is caused by a recent change that GitHub did a couple of months ago: https://github.blog/2021-09-01-improving-git-protocol-security-github/.

The git command seems to prefer the ED25519 key and only adds that to the known_hosts file (at least on my system)

ssh-keygen -F github.com
# Host github.com found: line 25 
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

However, go-git (via x/crypto/ssh) seems to prefer the ECDSA key. This is the key that gets passed to the HostKeyCallback: github.com:22 -> 140.82.121.3:22 -> ecdsa-sha2-nistp25nistp256AIHCc@G:ldd̻P}Kya23n9S.bMfv:hrH.

There's an issue open for this in go-git: go-git/go-git#411

So, the quick fix right now is to run ssh-keyscan github.com >> ~/.ssh/known_hosts first to add all GitHub keys to the known_hosts file. I have confirmed that this indeed works.

Although I would prefer not to do so, I'll add the option to switch off host key verification from Cerbos config if this problem does not get fixed upstream before the next release.

[meetme2meat]

@charithe ssh-keyscan github.com >> ~/.ssh/known_hosts this will out of scope in pod right?

[charithe]

I would do something like create a ConfigMap with the output of ssh-keyscan github.com and mount that at /etc/ssh/ssh_known_hosts for all Cerbos pods OR use an init container to do the same.

[charithe]

Come to think of it, in production, you probably don't want to distribute your SSH private key with the application anyway. Wouldn't it be easier to use HTTPS with a GitHub token instead?

[meetme2meat]

We thought about but for managing deploy key per repo easier as developer come and go it seem rather hard for us to work consistently with GITHUB token.

[meetme2meat]

I would do something like create a ConfigMap with the output of ssh-keyscan github.com and mount that at /etc/ssh/ssh_known_hosts for all Cerbos pods OR use an init container to do the same.

I'm testing that

[charithe]

We thought about but for managing deploy key per repo easier as developer come and go it seem rather hard for us to work consistently with GITHUB token.

I see. It's a shame GitHub token system is user-based. If I am not mistaken, deploy keys have admin rights don't they? Is creating an account for the system user in GitHub and generating a PAT with fewer privileges out of the question?

[meetme2meat]

@charithe

I would do something like create a ConfigMap with the output of ssh-keyscan github.com and mount that at /etc/ssh/ssh_known_hosts for all Cerbos pods OR use an init container to do the same

This worked

I see. It's a shame GitHub token system is user-based. If I am not mistaken, deploy keys have admin rights don't they? Is creating an account for the system user in GitHub and generating a PAT with fewer privileges out of the question?

But then it's extra headache IMO

[meetme2meat]

Closing this issue, @charithe thanks for help

[DataHearth]

Thanks for this ticket ! Both of you saved me so much time on this issue.

After updating my primary SSH key (and cleaned up my know_hosts) to the new GitHub standard, I was finally able to clone the repository without this error.