Sync clash with failed to diff ... git diff ... when using ManifestGeneratePaths and syncedRevision proceeds targetRevision

[toyamagu-2021]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

• ArgoCD sync clashed by follwoing error.
  • commit sha 481fbe3e687c8f339c0ecca000ee7ee7666a7aab proceeds 85dd68870f5c09225ae05c2b81e9136af7bce397.

ComparisonError: Failed to load target state: failed to compare revisions for source 1 of 1: rpc error: code = Internal desc = unable to get changed files for repo https://github.com/xxx/xxx.git with revision 85dd68870f5c09225ae05c2b81e9136af7bce397: failed to diff 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397: git diff --name-only 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397` failed exit status 128: fatal: Invalid revision range 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397 (retried 3 times).`

• We use manifestGeneratePaths with large monorepo and have multiple repo-server.

Root cause

• As is clear from the error message, the root cause is feat(controller): use manifest generate path during comparison (#14242) #15636

• argo-cd/reposerver/repository/repository.go

  Line 2778 in 27d1e64

  logCtx.Debugf("no changes found for application %s in repo %s from revision %s to revision %s", request.AppName, repo.Repo, syncedRevision, revision)

• Indirectly, feat(reposerver): Skip calling git fetch if commit to checkout exists locally #18657 cause this error.
• The issue may occurs following situation:
  • Someone commit s main branch with commit shaA.
  • Someone commits main branch with commit shaB, where changed files are not included in manifestGeneratePaths.
  • [Reconcilation Process] UpdateRevisionForPaths update cache key with commit shaB and change syncRevision to shaB.
  • Sync is queued with targetRevision shaA.
  • [SyncProcess] UpdateRevisionForPaths compares git diff shaB..shaA by syncOperation.
    • The local repository may not have commit sha B, so the error is thrown.
    • Note that git fetch origin is skipped in some cases by feat(reposerver): Skip calling git fetch if commit to checkout exists locally #18657

*To Reproduce

Described in root cause. Note that we need more than two repo-servers to reproduce this issue.

argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git cat-file -t db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=dde19
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git cat-file -t db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.7880369999999999
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git checkout --force db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=eae87
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git checkout --force db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=2.4994940000000003
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git clean -ffdx" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=5ae04
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.080456
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=8e750
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=error msg="`git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733` failed exit status 128: fatal: Invalid revision range c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" execID=8e750
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.462371
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to get changed files for repo https://github.com/toyamagu-2021/argo-cd-sandbox.git with revision db63a656360ed3a57c2bcd76f72ef38d793f5733: failed to diff c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733: `git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733` failed exit status 128: fatal: Invalid revision range c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" grpc.code=Internal grpc.method=UpdateRevisionForPaths grpc.service=repository.RepoServerService grpc.start_time="2024-12-01T16:20:41Z" grpc.time_ms=7.459 span.kind=server system=grpc

Expected behavior

Screenshots

Version

ArgoCD v2.13.1

Logs

Paste any relevant application logs here.

[pasha-codefresh]

Should we just not fail flow ( sync or reconcile ) in case if we were not able to update revision paths? i think it was mainly for performance optimization

[toyamagu-2021]

We can do that, but I think git fetch origin ${syncedRevision} is enough to avoid this corner case and unnecessary templating. #21015

[FangjianLu]

Thanks for the fix in #21015! We've been experiencing the error daily, is there a timeline to merge and release that?

[toyamagu-2021]

@pasha-codefresh
Hi. Can we merge #21015 ?
I've confirmed that cherry-piking this PR works fine in our environment.