Speed up git clones and catalog command #2893
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR started as a way to improve the speed of the
terragrunt catalog
command. However, I believe the improvements here may also speed up Terragrunt in general. Here's a summary of the changes:depth=1
parameter tosource
URLs. This will havego-getter
do a shallowgit clone
. Whenever we clone a repo, it is for read-only usage on a single branch/ref anyway, so a shallow clone should work fine, and should be much faster, especially for larger/older repos.git clone
steps of thecatalog
command concurrently using goroutines. So instead of cloning and parsing one repo at a time, we do them all in parallel (up to number of CPU cores), which speeds things up quite a bit as well.I tested the
catalog
command with 4 URLs in myterragrunt.hcl
config and the average time for first run when from ~20 seconds to ~5 seconds.TODOs
Read the Gruntwork contribution guidelines.
Release Notes (draft)
Added / Removed / Updated [X].
Migration Guide