Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up git clones and catalog command #2893

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

brikis98
Copy link
Member

Description

This PR started as a way to improve the speed of the terragrunt catalog command. However, I believe the improvements here may also speed up Terragrunt in general. Here's a summary of the changes:

  1. Add depth=1 parameter to source URLs. This will have go-getter do a shallow git clone. Whenever we clone a repo, it is for read-only usage on a single branch/ref anyway, so a shallow clone should work fine, and should be much faster, especially for larger/older repos.
  2. Run the git clone steps of the catalog command concurrently using goroutines. So instead of cloning and parsing one repo at a time, we do them all in parallel (up to number of CPU cores), which speeds things up quite a bit as well.

I tested the catalog command with 4 URLs in my terragrunt.hcl config and the average time for first run when from ~20 seconds to ~5 seconds.

TODOs

Read the Gruntwork contribution guidelines.

  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide

@brikis98
Copy link
Member Author

Ah, the scaffold command tries to look up Git tags on the repo, so shallow clone doesn't work... Will have to think on this a bit more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants