Unable to clone repository: entry not found

[cmfcruz]

Describe the bug

The source-controller is unable to clone the repository and its submodules. It returns the following error during each attempt: failed to checkout and determine revision: unable to clone '<repository_url>': entry not found.

We are using a Github repository that uses submodules. We use the HTTP URL of the repository for cloning the repository. We tried using both Github app installation tokens and personal access tokens for the referenced secret but it still fails with the same error message.

We can clone the repository and its submodules manually without any problems using the same Github app installation token or personal access tokens. We also haven't encountered this issue when using a repository that does not use submodules even while using the same referenced secret (Github app installation token or personal access token).

Please refer to the additional context section for the details of our GitRepository resource and some logs from the source-controller.

Steps to reproduce

1. Prepare a repository that uses submodules.
2. Use something like flux create secret git and set a Github app installation token or personal access token for cloning the repository.
3. Use something like flux create source git to reference the repository that uses submodules.
4. Attempt to reconcile the repository using flux reconcile source git.

Expected behavior

Flux source controller is able to clone the repository along with its submodules.

Screenshots and recordings

No response

OS / Distro

Ubuntu 22.04

Flux version

v2.2.2

Flux check

► checking prerequisites
✔ Kubernetes 1.28.4-eks-8cb36c9 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.2
✔ bootstrapped: false
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.3
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.3
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

Github

Container Registry provider

No response

Additional context

Our git repository resource:

$ kubectl describe gitrepository -n flux-system flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  reconcile.fluxcd.io/requestedAt: 2024-01-24T20:45:12.151253713+08:00
API Version:  source.toolkit.fluxcd.io/v1
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2024-01-24T12:41:55Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  17388672
  UID:               da0abf10-563f-47f0-9536-51ab541e4667
Spec:
  Interval:            10m0s
  Recurse Submodules:  true
  Ref:
    Branch:  master
  Secret Ref:
    Name:   flux-system
  Timeout:  60s
  URL:      https://github.com/product-os/environment-playground.git
Status:
  Conditions:
    Last Transition Time:     2024-01-24T12:52:03Z
    Message:                  building artifact
    Observed Generation:      1
    Reason:                   ProgressingWithRetry
    Status:                   True
    Type:                     Reconciling
    Last Transition Time:     2024-01-24T12:52:03Z
    Message:                  failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   False
    Type:                     Ready
    Last Transition Time:     2024-01-24T12:42:00Z
    Message:                  failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   True
    Type:                     FetchFailed
  Last Handled Reconcile At:  2024-01-24T20:45:12.151253713+08:00
  Observed Generation:        -1
Events:
  Type     Reason              Age                   From               Message
  ----     ------              ----                  ----               -------
  Warning  GitOperationFailed  5m14s (x10 over 15m)  source-controller  failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found

Some logs from the source-controller (from start-up to recurring error message):

kubectl logs -n flux-system deployment/source-controller
{"level":"info","ts":"2024-01-24T12:26:39.337Z","logger":"setup","msg":"caching of Helm index files is disabled"}
{"level":"info","ts":"2024-01-24T12:26:39.337Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-01-24T12:26:39.337Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-01-24T12:26:39.337Z","msg":"starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","ts":"2024-01-24T12:26:39.337Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-01-24T12:26:39.438Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/source-controller-leader-election..."}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","logger":"runtime","msg":"successfully acquired lease flux-system/source-controller-leader-election"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","source":"kind source: *v1.GitRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting Controller","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.HelmChart"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.HelmRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1.GitRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting Controller","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","source":"kind source: *v1beta2.HelmRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting Controller","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository","source":"kind source: *v1beta2.OCIRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting Controller","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting EventSource","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","msg":"Starting Controller","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket"}
{"level":"info","ts":"2024-01-24T12:26:39.451Z","logger":"setup","msg":"starting file server"}
{"level":"info","ts":"2024-01-24T12:26:39.551Z","msg":"Starting workers","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","worker count":2}
{"level":"info","ts":"2024-01-24T12:26:39.551Z","msg":"Starting workers","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","worker count":2}
{"level":"info","ts":"2024-01-24T12:26:39.551Z","msg":"Starting workers","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","worker count":2}
{"level":"info","ts":"2024-01-24T12:26:39.552Z","msg":"Starting workers","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository","worker count":2}
{"level":"info","ts":"2024-01-24T12:26:39.552Z","msg":"Starting workers","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","worker count":2}
{"level":"error","ts":"2024-01-24T12:26:45.071Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"bc7f7d07-4021-4add-a517-12012cf471a1","error":"failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found"}
{"level":"error","ts":"2024-01-24T12:26:50.267Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"1acdefef-f082-4a5c-aa7e-f3d155a92bbb","error":"failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found"}
{"level":"error","ts":"2024-01-24T12:26:56.175Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"ef5da1ee-2220-41ef-bef3-8dbec98a54b1","error":"failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found"}
{"level":"error","ts":"2024-01-24T12:26:59.896Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"858f27e4-59ec-494a-b65e-5ec426c5dfdd","error":"failed to checkout and determine revision: unable to clone 'https://github.com/product-os/environment-playground.git': entry not found"}
...

Code of Conduct

• I agree to follow this project's Code of Conduct