argocd cli throws ssh handshake error on app sync

[fivesheep]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When running argocd command line from desktop to sync an project, it gave the following error:

argocd app sync my-project
ERRO[0001] Could not set-up SSH known hosts callback: open /app/config/ssh/ssh_known_hosts: no such file or directory
FATA[0001] rpc error: code = FailedPrecondition desc = ssh: handshake failed: knownhosts: key mismatch

Likely related to #7723. However, we had already fix update the known ssh cm as well as created a new key with ecdsa, as well as upgraded argocd server to the latest 2.1.7 version. We had also validated that the solution works well when during auto-sync and triggering sync from the Web UI that is running on the cluster.

To Reproduce

kubectl config set-context --current --namespace=argocd

argocd login --core

argocd app sync my-app

A project with private github repo is configured for the app, and an ssh deploy key is used.

Expected behavior

It shall just trigger the sync like clicking sync from the Web UI

Version

argocd: v2.1.7+a408e29
  BuildDate: 2021-11-17T22:12:41Z
  GitCommit: a408e299ffa743213df3aa9135bf7945644ec936
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.1.7+a408e29
  BuildDate: 2021-11-17T22:12:41Z
  GitCommit: a408e299ffa743213df3aa9135bf7945644ec936
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: darwin/amd64
  Ksonnet Version: unable to determine ksonnet version: exec: "ks": executable file not found in $PATH
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

same error on 2.1.6

Logs

ERRO[0001] Could not set-up SSH known hosts callback: open /app/config/ssh/ssh_known_hosts: no such file or directory
FATA[0001] rpc error: code = FailedPrecondition desc = ssh: handshake failed: knownhosts: key mismatch

[jannfis]

Hi. In an upgrade scenario, you need to update the manifests as well, since they host keys are stored in the argocd-known-hosts-cm ConfigMap, not in the container image.

[jannfis]

Oh. Sorry, I misunderstood the problem. Sorry, my fault.

@alexmt This is actually about core feature not having access to the known hosts.

[alexmt]

Thank you for pointing this out @jannfis ! Now I've realized this is a duplicate of #7565 . @ishitasequeira started working on PR #5832 that fixes it . PR looks pretty close to completion!