Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable SSH agent support on Windows #1008

Merged
merged 2 commits into from
Apr 19, 2024

Conversation

shanman190
Copy link
Contributor

@shanman190 shanman190 commented Apr 16, 2024

The purpose of this PR is to enable SSH agent support universally throughout DevPod. As mentioned in #875 the upstream golang.org/x/crypto/ssh/agent library doesn't support named pipes and only supports unix sockets. This results in any usage requiring SSH agent forwarding -- outside of using the native SSH client directly -- not being enabled, even if requested.

While it would be ideal for the upstream library to incorporate named pipes support, it's been frozen in place since early 2021 awaiting the inclusion of golang.org/x/sys/windows/namedpipe (via golang/go#49650). However, the aforementioned issue has been slow in getting adopted due to Microsoft copyright review due to some included components on that PR in particular.

As a result of all of this, this PR introduces a thin shim that coordinates SSH Agent forwarding between the upstream golang.org/x/crypto/ssh library for unix systems and a mirror implementation of the golang SSH agent support, but using named pipes.

@shanman190
Copy link
Contributor Author

For the why is this important, presently without SSH agent forwarding working on Windows this means that DevPod can't seamlessly provide transitive identity into the machine or workspace using it's native implementation. This results in being unable to perform SSH-based Git commands against repositories and being forced to fallback to either:

  • Workspace Init: a local clone, then DevPod copies the local content to the container (This typically results in EOL issues as Windows clones CRLF by default)
  • Ongoing Git operations: the user is either forced to setup an existing or new SSH key within the container to then perform SSH operations OR use an alternative authentication method.

@shanman190
Copy link
Contributor Author

Also there were two more places where SSH_AUTH_SOCK appears, but it wasn't super clear where those invocations originate from, so for the purposes of this change I left those unmodified.

Comment on lines +19 to +29
func GetSSHAuthSocket() string {
sshAuthSocket := os.Getenv("SSH_AUTH_SOCK")
if sshAuthSocket != "" {
return sshAuthSocket
}
if _, err := os.Stat(defaultNamedPipe); err == nil {
return defaultNamedPipe
}

return ""
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm planning to test this just a little bit more tomorrow, but presently I know this works with the OpenSSH agent that ships with Windows 10/11 natively. I'm pretty sure that the Git for Windows variation utilizes a unix socket which would mean a slight adjustment to the agent forwarding to ensure the correct socket type is used for the connection.

More specifically, the OpenSSH build that comes built into Windows 10/11 creates a named pipe at \\.\pipe\openssh-ssh-agent which is what I've defined as a constant above here. This pipe is hardcoded in the Microsoft OpenSSH build and is used by default when connecting to the agent.

The Git for Windows build on the other hand is creating its files in the TEMP directory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've confirmed that Git for Windows is in fact using an AF_UNIX socket, so I'll update the forward method just a little bit to handle that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some more digging after being puzzled why I couldn't connect using the AF_UNIX socket type as directly utilized by the upstream golang.org/x/crypto/ssh/agent, I've come to find out that MSYS2 which Git for Windows is based on uses an emulated socket rather than a native AF_UNIX socket as I was led to believe.

Now with this said, there are a myriad of authenticators that do create valid AF_UNIX sockets for Windows and I found a PR for Win32-OpenSSH to add native AF_UNIX sockets to the build that comes packaged with Windows. Unfortunately, that PR is not yet merged.

What I think I'm going to do here is to prefer a set SSH_AUTH_SOCK, this way various correct implementations work, then fallback to the named pipe implementation, then finally return an error. When Windows finally natively supports SSH_AUTH_SOCK definitions, then the code will natively migrate to that updated path this way.

I'll submit this small change tomorrow, but then this should be good to go.

@pascalbreuninger
Copy link
Member

@shanman190 thanks for the PR, we'll look into it soon

@pascalbreuninger
Copy link
Member

@shanman190 first up, thanks a lot for your work on this issue and the detailed explanation!
Both @89luca89 and me aren't Windows users and aren't really familiar with the environment, we appreciate all the help we can get there.

Also there were two more places where SSH_AUTH_SOCK appears, but it wasn't super clear where those invocations originate from, so for the purposes of this change I left those unmodified.

I took a look and they are inside of the container, should be fine 👍

@shanman190
Copy link
Contributor Author

shanman190 commented Apr 19, 2024

@pascalbreuninger, you're welcome! Thanks to both you and @89luca89 taking a look as well. I think that I've got everything here working in a way that should work out very nicely for us all. 😄

I do also think that this should address at least most of the issues related to Windows and SSH as well, so we might have those users retest once the release has been cut.

Copy link
Member

@pascalbreuninger pascalbreuninger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🙌

@pascalbreuninger pascalbreuninger merged commit 0852c89 into loft-sh:main Apr 19, 2024
25 checks passed
@shanman190 shanman190 deleted the feature/ssh-agent branch April 19, 2024 19:12
@shanman190
Copy link
Contributor Author

shanman190 commented Apr 24, 2024

@pascalbreuninger, so everything here looks great on the latest alpha. One quick question though that I have with ssh clones still failing is that I'm seeing it clone with the Git credential helper enabled even though my AWS provider has it explicitly disabled. So far I haven't been able to come up with any set of combinations that will actually disable it. Do you have any ideas?

On the flip side, it looks like the inject docker credentials option changes as expected when I update the AWS provider configuration value. Just for some reason not the inject Git credentials option.

@pascalbreuninger
Copy link
Member

@shanman190 great 👍

Maybe try the context option devpod context set-options -o SSH_INJECT_GIT_CREDENTIALS=false
This is a problem of specificity (user option > provider option). I'm curious, would you expect this to be the other way round and if so, why?
ResolveAgentConfig

@shanman190
Copy link
Contributor Author

@pascalbreuninger, so based on

options := provider2.ToOptions(workspace, machine, devConfig.ProviderOptions(provider.Name))
it feels kind of backwards.

In my mind the precedence kinda feels like it should be:

  • (lowest) provider defaults (SSH, AWS, Kubernetes, etc)
  • user preferences ~/.devpod/config.yaml
  • machine, if applicable
  • (highest) workspace

I think that's just an illustration of the same thing that you said though, but please correct me if I'm wrong.

To me though in the order that I've laid out it's furthest away from the workspace to the closest. The closer to the workspace the setting is the more prevalent I would think that option would become.

For my specific setup, I'm setting INJECT_GIT_CREDENTIALS and INJECT_DOCKER_CREDENTIALS,

  • Provider - AWS: git=true, docker=true
  • User: git=false, docker=false
  • Machine: git=false, docker=false (looks like it cloned the settings from the user config when the machine was first created)
  • Workspace: none configured

NOTE: agent.injectGitCredentials as a setting only appears in the AWS provider.yaml, but since it's using the property placeholder I would think that it's resolution would be delayed as late as possible.

So I would envision the machine level taking precedence, even though I didn't explicitly set those (the machine was created as a side effect of creating the workspace in the DevPod UI), resulting in those settings being false when creating the workspace.

@shanman190
Copy link
Contributor Author

Alright, I've discovered that the helper isn't my problem. For whatever reason, I was missing that DevPod sudo's to root where it then clones. sudo doesn't take the environment variables by default, but it can easily be told to do so. So I'll have another fix incoming that should finalize this feature and actually make ssh clones work everywhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants