fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys #1126

gabriel-farache · 2023-12-22T11:05:39Z

When I try to use ssh key to clone the git repo:

move2kube plan -s git+ssh://bitbucket.org/gfarache31/m2k-test@master

I get the following error:

FATA[0000] failed to create the plan. Error: "failed to clone the repo 'git+ssh://bitbucket.org/gfarache31/m2k-test'. Error: failed to clone using vcs url 'git+ssh://bitbucket.org/gfarache31/m2k-test' and clone options {CommitDepth:1 Overwrite:true MaxSize:-1 CloneDestinationPath:/tmp/move2kube1491402545/m2ksources}. Error: failed to perform clone operation using git with options {URL:git@bitbucket.org:gfarache31/m2k-test Auth:<nil> RemoteName:origin ReferenceName:refs/heads/main SingleBranch:true Mirror:false NoCheckout:false Depth:1 RecurseSubmodules:0 Progress:<nil> Tags:2 InsecureSkipTLS:false CABundle:[] ProxyOptions:{URL: Username: Password:}} and {CommitDepth:1 Overwrite:true MaxSize:-1 CloneDestinationPath:/tmp/move2kube1491402545/m2ksources}. Error: error creating SSH agent: \"SSH agent requested but SSH_AUTH_SOCK not-specified\""

If in install openssh I run the following, it works:

eval $(ssh-agent -s)
<create pub and priv keys in .ssh>
ssh-add .ssh/id_rsa
ssh-keyscan -t rsa bitbucket.org > .ssh/known_hosts
move2kube plan -s git+ssh://bitbucket.org/gfarache31/m2k-test@master

So in order to fix this, I add the following in the Dockerfile:

RUN microdnf install -y openssh openssh-clients
RUN mkdir -p $HOME/.ssh
RUN printf "bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==\ngitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf\ngitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9\ngitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=\nbitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==" > $HOME/.ssh/known_hosts

The values added in the known_hosts file are copied from common/sshkeys/sshkeys.go

github-actions · 2023-12-22T11:05:52Z

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

codecov · 2023-12-22T11:12:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 14.85%. Comparing base (ff13fa4) to head (888415e).

❗ Current head 888415e differs from pull request most recent head 2885948. Consider uploading reports for the commit 2885948 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1126   +/-   ##
=======================================
  Coverage   14.85%   14.85%           
=======================================
  Files          90       90           
  Lines        8379     8379           
=======================================
  Hits         1245     1245           
  Misses       6813     6813           
  Partials      321      321

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gabriel-farache · 2023-12-22T13:48:27Z

RUN printf "bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==\ngitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf\ngitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9\ngitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=\nbitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==" > $HOME/.ssh/known_hosts

could be replaced by calls to ssh-scan for each of the 3 domains, should I do that instead?

HarikrishnanBalagopal · 2024-01-04T07:13:26Z

Dockerfile

+
+RUN microdnf install -y openssh openssh-clients
+RUN mkdir -p $HOME/.ssh
+RUN printf "bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==\ngitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf\ngitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9\ngitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=\nbitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==" > $HOME/.ssh/known_hosts


Can you please explain this part? Do we really need to hardcode the public keys?

could be replaced by calls to ssh-scan for each of the 3 domains, should I do that instead?

Even if we do that, the public keys will be hardcoded in that image forever at the time of creation docker build.

Those public keys are hard coded in the codebase, in the spirit of consistency I added them there too but I am more than happy to remove them :)

@kmehant Do we need to install ssh ? I think we use a library to do the cloning

move2kube/common/vcs/git.go

Line 29 in aa79dfa

"github.com/go-git/go-git/v5"

Related:

Error: ssh: handshake failed: knownhosts: key mismatch go-git/go-git#619

ssh: unable to ignore host key auth callback go-git/go-git#654

git clone gives error with SSH Agent go-git/go-git#143

@gabriel-farache Does this allow cloning from Github Enterprise? like github.mycompany.com ?
I think most often the private repos will be in a private company wide Github/GitLab.

#1126 (comment)

go-git library internally can pick up the SSH agent on the host/container for remote repository pulling.

@kmehant Well, I am using the latest image from quay and I got the following error:

$ move2kube plan -s git+ssh://bitbucket.org/gfarache31/m2k-test INFO[0000] Cloning the repository using git into '/tmp/move2kube2496147635/m2ksources' . This might take some time. FATA[0000] failed to create the plan. Error: "failed to clone the repo 'git+ssh://bitbucket.org/gfarache31/m2k-test'. Error: failed to clone using vcs url 'git+ssh://bitbucket.org/gfarache31/m2k-test' and clone options {CommitDepth:1 Overwrite:true MaxSize:-1 CloneDestinationPath:/tmp/move2kube2496147635/m2ksources}. Error: failed to perform clone operation using git with options {URL:git@bitbucket.org:gfarache31/m2k-test Auth:<nil> RemoteName:origin ReferenceName:refs/heads/main SingleBranch:true Mirror:false NoCheckout:false Depth:1 RecurseSubmodules:0 Progress:<nil> Tags:2 InsecureSkipTLS:false CABundle:[] ProxyOptions:{URL: Username: Password:}} and {CommitDepth:1 Overwrite:true MaxSize:-1 CloneDestinationPath:/tmp/move2kube2496147635/m2ksources}. Error: error creating SSH agent: \"SSH agent requested but SSH_AUTH_SOCK not-specified\""

How did you have that working for you?

@HarikrishnanBalagopal I'll remove the Dockerfile entry for the known hosts

#1126 (comment)

@gabriel-farache I have not tested it, but, as I mentioned here (#1126 (comment)) go-git would pick up ssh agent from the host/container and thanks for adding openssh to the Dockerfile so go-git should now have an ssh agent to pick up.

@kmehant if it's tested and working we can merge this PR.

@HarikrishnanBalagopal just tested it, we need to start the ssh-agent and init the .ssh/ keys.

For which we need to run

eval "$(ssh-agent)" && ssh-add

This should go after spawning the container with .ssh folder mounted, so should we be adding that to docs somewhere? if so let me merge this PR.

kmehant · 2024-01-28T17:03:08Z

@gabriel-farache can you fetch and rebase your PR branch? Thank you.

gabriel-farache · 2024-01-29T11:12:55Z

@gabriel-farache can you fetch and rebase your PR branch? Thank you.

@kmehant done :)

kmehant

@gabriel-farache Thank you for the contribution!

Signed-off-by: gabriel-farache <gfarache@redhat.com>

kmehant · 2024-03-13T06:57:24Z

@gabriel-farache Thank you for the contribution. I want to confirm, on top of the dependencies that you have added, while running the container mounting .ssh folder having public, private keys, and knownhosts file and using CMD to start ssh agent and perform ssh-add on the mounted key would suffice your usecase, is it?

gabriel-farache · 2024-03-13T07:50:54Z

@kmehant

@gabriel-farache Thank you for the contribution. I want to confirm, on top of the dependencies that you have added, while running the container mounting .ssh folder having public, private keys, and knownhosts file and using CMD to start ssh agent and perform ssh-add on the mounted key would suffice your usecase, is it?

Yes it fulfill my needs for our use case.

Thank you for merging :)

gabriel-farache changed the title ~~fix install openssh in Dockerfile in order to be able to clone repo using ssh keys~~ fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys Dec 22, 2023

github-actions bot added the fix label Dec 22, 2023

HarikrishnanBalagopal reviewed Jan 4, 2024

View reviewed changes

gabriel-farache mentioned this pull request Jan 4, 2024

move2kube workflow using ssh keys instead of token parodos-dev/serverless-workflow-examples#15

Merged

gabriel-farache force-pushed the fix/add_openssh branch from c76052f to 383bfd6 Compare January 8, 2024 15:08

kmehant requested a review from HarikrishnanBalagopal January 28, 2024 15:21

gabriel-farache force-pushed the fix/add_openssh branch from 9563543 to d0ffac3 Compare January 29, 2024 11:12

gabriel-farache requested a review from kmehant March 12, 2024 09:15

kmehant approved these changes Mar 13, 2024

View reviewed changes

fix: install openssh in order to be able to clone repo using ssh keys

2885948

Signed-off-by: gabriel-farache <gfarache@redhat.com>

kmehant force-pushed the fix/add_openssh branch from 888415e to 2885948 Compare March 13, 2024 06:56

kmehant enabled auto-merge (squash) March 13, 2024 06:56

kmehant merged commit fc28c23 into konveyor:main Mar 13, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys #1126

fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys #1126

gabriel-farache commented Dec 22, 2023

github-actions bot commented Dec 22, 2023

codecov bot commented Dec 22, 2023 •

edited

gabriel-farache commented Dec 22, 2023

HarikrishnanBalagopal Jan 4, 2024 •

edited

gabriel-farache Jan 4, 2024

HarikrishnanBalagopal Jan 4, 2024 •

edited

HarikrishnanBalagopal Jan 5, 2024 •

edited

kmehant Jan 5, 2024

gabriel-farache Jan 8, 2024

gabriel-farache Jan 8, 2024

kmehant Jan 28, 2024 •

edited

HarikrishnanBalagopal Jan 28, 2024

kmehant Jan 29, 2024 •

edited

kmehant commented Jan 28, 2024

gabriel-farache commented Jan 29, 2024

kmehant left a comment

kmehant commented Mar 13, 2024

gabriel-farache commented Mar 13, 2024

fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys #1126

fix: install openssh in Dockerfile in order to be able to clone repo using ssh keys #1126

Conversation

gabriel-farache commented Dec 22, 2023

github-actions bot commented Dec 22, 2023

codecov bot commented Dec 22, 2023 • edited

Codecov Report

gabriel-farache commented Dec 22, 2023

HarikrishnanBalagopal Jan 4, 2024 • edited

Choose a reason for hiding this comment

gabriel-farache Jan 4, 2024

Choose a reason for hiding this comment

HarikrishnanBalagopal Jan 4, 2024 • edited

Choose a reason for hiding this comment

HarikrishnanBalagopal Jan 5, 2024 • edited

Choose a reason for hiding this comment

kmehant Jan 5, 2024

Choose a reason for hiding this comment

gabriel-farache Jan 8, 2024

Choose a reason for hiding this comment

gabriel-farache Jan 8, 2024

Choose a reason for hiding this comment

kmehant Jan 28, 2024 • edited

Choose a reason for hiding this comment

HarikrishnanBalagopal Jan 28, 2024

Choose a reason for hiding this comment

kmehant Jan 29, 2024 • edited

Choose a reason for hiding this comment

kmehant commented Jan 28, 2024

gabriel-farache commented Jan 29, 2024

kmehant left a comment

Choose a reason for hiding this comment

kmehant commented Mar 13, 2024

gabriel-farache commented Mar 13, 2024

codecov bot commented Dec 22, 2023 •

edited

HarikrishnanBalagopal Jan 4, 2024 •

edited

HarikrishnanBalagopal Jan 4, 2024 •

edited

HarikrishnanBalagopal Jan 5, 2024 •

edited

kmehant Jan 28, 2024 •

edited

kmehant Jan 29, 2024 •

edited