[SURE-7410] Git Clone Error: Empty git-upload-pack-given using SSH or HTTPS to Azure DevOps Repos

[zbutler-trellance]

Rancher Server Setup

• Rancher version: 2.8

Information about the Cluster

• Kubernetes version: 1.25.15
• Cluster Type: Local
• Hosted: AKS

Describe the bug
Rancher Fleet jobs report: "Error cloning repository: empty git-upload-pack given" when connecting to a previously working Git Repo in Azure DevOps Repos using both SSH and HTTPS authentication.

To Reproduce

• Upgraded Rancher 2.7.9 to Rancher 2.8

Result

• Git Repos are stuck in the status Updating Git

Expected Result

• Git Repos sync like they did in 2.7.9

Screenshots

[matteogazzadi]

Same here. Any new GitRepo created after update to 2.8.0 fails if connecting to Azure DevOps

[kkaempf]

Does a manual clone (using git) still work ?

[kkaempf]

src-d/go-git#1058go-git/go-git#64 ?!

[raulcabello]

src-d/go-git#1058go-git/go-git#64 ?!

That's the problem. I already have a fix for this issue. However, I'd like to do more testing with azure before submitting a PR. I'm currently working on #1974 that also involves azure devops

[wagneripjr]

I need this fix to start using Fleet. 🥲
The first implementation of one gitrepo with helm/kustomize is stuck because the gitjob can't download the repository from Azure Devops.

I've tested the fix locally, and it worked.

Do you think I can help in any way? 🙏

[raulcabello]

@wagneripjr thanks for testing the fix! It's also working in my local tests.
I just created a PR with the fix.

[wagneripjr]

Thanks @raulcabello
Meanwhile, I've built an image of gitjob in my private registry with your version of gitcloner and modified Fleet using helm upgrade to change the gitjob image.

helm values.yaml:
. . .
gitjob:
priorityClassName: rancher-critical
gitjob:
repository: private-registry/gitjob
tag: azure
. . .

It worked well, but now I'm facing another issue, but it is unrelated; my gitrepo has kustomize and helm. Fleet shows everything is applied to my cluster, but in fact, only kustomize was applied; all other objects related to the helm chart are part of the helm bundle but don't exist in my cluster 🤔

[raulcabello]

/forwardport v2.9.0

[raulcabello]

QA Template

Solution

Azure DevOps requires capabilities multi_ack / multi_ack_detailed, which are not fully implemented and by default are included in transport.UnsupportedCapabilities. transport.UnsupportedCapabilities have been changed to support that.

Testing

• Test GitRepos are deployed successfully for git repositories hosted in Azure DevOps.
• All the other git providers should keep working fine (e.g. github, bitbucket, gitlab)

[raulcabello]

Public repos in Azure DevOps can't be cloned with the fix from this PR
This will be fixed in go-git. In the meantime, this fix will allow users to clone private repos in Azure DevOps

[femtonelson]

Same error here. I had to downgrade Fleet to "fleet:102.2.1+up0.8.1" for it to work.
URL format : https://dev.azure.com/xxx-organization-xxx/xxx-project-xxx/_git/xxx-repo-xxx
username : any
password : PAT

Kubernetes version: 1.26.10
Cluster Type: Local
Hosted: AKS

[thehejik]

Test report

Works as expected. I could create a gitRepos on both workspaces (fleet-local, fleet-default) by using Private Azure DevOps git repos over HTTPS (BASIC AUTH) and SSH (keys) URLs. The repos were cloned and its resources provisioned successfully on both clusters.

Environment

Tested on local k3s with v1.27.10+k3s2 and downstream k3s with v1.28.6+k3s2 on rancher:v2.8.3-rc1 with fleet App 103.1.1+up0.9.1-rc.5 using following images:

rancher/fleet:v0.9.1-rc.5
rancher/gitjob:v0.9.3
rancher/fleet-agent:v0.9.1-rc.5

[kkaempf]

@thehejik - please close the issue when all tests have passed

[zbutler-trellance]

Sweet! :) Thanks!

[ericksilvaRendimento]

@femtonelson Can you please, tell me how to downgrade to this version?

Same error here. I had to downgrade Fleet to "fleet:102.2.1+up0.8.1" for it to work.

I tried using helm commands but fleet repo lists app 0.9.0 with chart 0.9.0. How to specify this chart version ?

[femtonelson]

@ericksilvaRendimento In Rancher Manager UI, go to : local cluster -> Apps -> Installed Apps -> fleet -> Edit/Upgrade -> Version