Unable to create application from a connected Azure DevOps server repository

[nilic]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After not being able to connect a private Azure DevOps server repository to ArgoCD via username and password (with error returned "Unable to connect HTTPS repository: authentication required)", I have successfully connected the repo via Personal Access Token (PAT):

When I try to create an application using the repository, I get the following error:

Unable to create application: application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch 493dd952d54b8817daa972adc056d32a1f31231a: git fetch origin --tags --force failed exit status 128: fatal: Authentication failed for '<redacted>/_git/argocd-apps/'

PAT used has Full Access scope and works with git operations to the repo, which was tested locally with git clone and git fetch origin --tags --force which is reported in the ArgoCD error message.

My guess is that this is related to Azure DevOps not allowing PAT authentication with a non-empty username (tested locally with git), while ArgoCD requires username to be a non-empty string when adding repository credentials. ArgoCD is probably passing this username to Azure DevOps which results in the mentioned error. Related issue: #6098

To Reproduce

• Create a repository on Azure Devops server
• Create an Azure DevOps server Personal Access Token (PAT) with Full Access scope
• Add the repository to ArgoCD via HTTPS using the PAT
• Try to create an ArgoCD application using the added repository

Expected behavior

ArgoCD application is successfully created.

Screenshots

Version

argocd-server: v2.0.0+f5119c0
  BuildDate: 2021-04-07T06:00:33Z
  GitCommit: f5119c06686399134b3f296d44445bcdbc778d42
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

[kshamajain99]

@nilic This error is from github directly. While accessing a repo, github returns fatal: Authentication failed error if username is not provided.

$ git clone https://github.com/username/Private-Repo-Test.git
Cloning into 'Private-Repo-Test'...
Username for 'https://github.com':
Password for 'https://github.com': <personal-access-token>
remote: Repository not found.
fatal: Authentication failed for 'https://github.com/kshamajain99/Private-Repo-Test.git/'

$ git clone https://github.com/username/Private-Repo-Test.git
Cloning into 'Private-Repo-Test'...
Username for 'https://github.com': fakeusername
Password for 'https://fakeusername@github.com': <personal-access-token>
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 9 (delta 0), reused 6 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), done.

[nilic]

I'm not sure how this is related to my issue, since I'm not using GitHub, but Azure DevOps server. Although I agree that this seems to be a backend-specific problem.

After reading the following on authenticating to Azure DevOps through HTTPS using PAT:

• Azure DevOps documentation - Use personal access tokens

• Azure DevOps: No example on how to use PAT to access git repositories

I've decided to workaround the issue by adding the Azure DevOps repository via SSH, in which case new app creation works fine.

[rouke-broersma]

@nilic we have successfully used azure devops with argo by creating the PAT from the clone window. There it specifies the exact username to use. This does work, random username does not:

[nilic]

That's interesting - it seems that this is an Azure DevOps Services feature, since I don't have that option in Azure DevOps Server 2020:

I have created the PAT as described here and no username was given. Here are some findings:

• git v2.9.2.windows.1: I can git clone the repo using empty username and PAT (in clear text, not base64 encoded). Non-empty username is a no go.
• git v2.18.4: I am unable to git clone the repository using any combination of non-base64 encoded username and PAT (tried with empty username, my AzDO username with the domain name, my AzDO username without the domain name, random string, event the token name). The only way to clone is as described here.

So it's seems the behavior also depends on the git version (possibly also on the OS?). I'd say that currently the safest way to use AzDO Server with ArgoCD is by adding the repo via SSH.

[lucheng-wh]

I met same problem with nilic

[bdzanko17]

I am having the same problem, did someone figured this out?

[AtzeDeVries]

So if you create a PAT for a user (not just a PAT for a specific repo, which is only available in non azdo server) and then create a random username it works.

---
apiVersion: v1
kind: Secret
metadata:
  name: azdo-<redacted>-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: https://dev.azure.com/<redacted>
  password: <redacted>
  username: argocd

[fr3man1]

@AtzeDeVries Does this work on AzDO Server?

[rouke-broersma]

@AtzeDeVries Does this work on AzDO Server?

We could not get this working on azdo server because apparently azdo server requires some git extensions that the go got client does not support.

Only works on azdo cloud.

[luber]

Any updates on this? Did anyone find any workarounds?
We are running Azure DevOps Server 2020 Update 1.1. But the same issue here.

Looks like MS is the only who requires multi-ack functionality and go-git can't provide it.
plumber-cd/terraform-backend-git#15 (comment)

[willbush]

Did anyone find any workarounds?

@luber I think using ssh. However, expect problems.

Firstly, with on-prem hosting. You're going to run into: https://argo-cd.readthedocs.io/en/release-2.0/user-guide/private-repositories/#unknown-ssh-hosts

Fix for me by doing:

ssh-keyscan your-server-host-name | argocd cert add-ssh --batch

Then, because ArgoCD removed support for SHA-1 signature hash algorithm and Azure Devops still seems to be on SHA-1 [1] [2]

Think you may need to modify argocd install to support signature hash algorithm AzDO supports sort of like this #6812 (comment)

I got to this point and we decided to start migrating off Azure Devops to Github instead because we doubt Microsoft is going to continue to support Azure Devops on the same level as Github in the future.

[nilic]

For Azure DevOps Server 2020 Update 1.1 this workaround is needed: #6812 (comment), just replace ssh.dev.azure.com with the name of your Azure DevOps server and remove line User git.

Azure DevOps Server 2022 RC2 adds support for additional signature algorithms besides ssh-rsa so hopefully this workaround won't be necessary after upgrading to the new version - Azure DevOps Server 2022 RC2 Release Notes

[nilic]

I have tested ArgoCD v2.5.0+a765c14 with Azure DevOps Server 2022 RC2 and can confirm that the ssh-rsa workaround is no longer needed.

[andrescolodrero]

@nilic Hi, do you have the instruccions to connect to Azure devops? IM starting to get lost with many
I have both AZDO on prem (2020) and cloud, and trying connection from https or git, I´d rather prefer to setup https than SSH

[rouke-broersma]

@nilic Hi, do you have the instruccions to connect to Azure devops? IM starting to get lost with many
I have both AZDO on prem (2020) and cloud, and trying connection from https or git, I´d rather prefer to setup https than SSH

Azdo on prem 2020 does not work, if you upgrade to 2022 it will work.

[nilic]

AFAIK, AzDO Server (aka on-prem) can't work with HTTPS, only SSH. AzDO Server 2020 works as described in #6362 (comment), while AzDO Server 2022 doesn't need that workaround (works out of the box).

A few additional notes on adding AzDO Server to ArgoCD via SSH:

• you need to add SSH fingerprint of your AzDO Server to ArgoCD Settings - Certificates - Add SSH known hosts; fingerprint needs to be in the known_hosts format and the easiest way to get this is by running ssh-keyscan <your AzDO Server address>
• repository is added to ArgoCD without the ssh:// prefix
• repo connection status of "Successful" in ArgoCD doesn't necessarily mean that everything is fine - you need to try to create an ArgoCD app from the repo; I've had "successful" repos failing when ArgoCD tries to pull from the repo;
• to troubleshoot adding the repo to ArgoCD you can use the logs of argocd-repo-server pod

I don't use Azure DevOps Services (aka cloud) so can't comment on its interoperability with ArgoCD.

[rouke-broersma]

Azure devops services is fully working