Terraform Lockfile Missing zh Hashes

[ekristen]

How are you running Renovate?

WhiteSource Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

Please select which platform you are using if self-hosting.

github.com

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Describe the bug

The terraform manager only calculates the h1 hashes but zh hashes are still used and expected (even though they are legacy), if you then subsequently run terraform init on at least terraform 1.1 it'll modify the lockfile with zh hashes, this then results in inconsistent lockfile errors depending on when and how this is run.

It would be best until zh is fully deprecated to calculate and add those hashes to the lockfile entries too.

Relevant debug logs

Logs

Copy/paste any log here, between the starting and ending backticks

Have you created a minimal reproduction repository?

No reproduction repository

[viceice]

Use terraform init -lockfile=readonly to init, so lockfile won't touched

[ekristen]

@viceice while that will work, I would consider that a workaround. If renovate is going to modify the lockfile, it should modify it correctly, terraform expects the zh hashes to be added and doesn't provide a way to NOT add them with the terraform binary, so renovate should act the same.

[viceice]

We discussed this already in discussion #11821 and it was declined

[ekristen]

@viceice thanks, I'll go jump on that discussion as well. I disagree with the authors opinion on this. No need to duplicate my comments though, but I will link back.

[ekristen]

My comments are here #11821 (comment)

[viceice]

ref:

• Terraform lock files #7895
• Add CLI command to upgrade lock file without full initialization hashicorp/terraform#27161

[rarkins]

Thanks. From that I can see that running terraform init is the only way to update a lock file, but it has a few failure scenarios which are likely to happen to Renovate users and prevent a successful update. Therefore we can continue with "hand crafting" the lockfile update as long as it's feasible.

[ekristen]

Another possible issue is that it appears that the constraints are performed as terraform would create them.

For example when renovate updates a constraint it appears to be something like ~> 2.2.0 but if you run terraform, it'll do a larger swath of versions based on nested dependencies or at least it seems, for example >= 2.0.0, >=2.2.0, ~> 2.2.0

I'm not sure exactly why, but this seems to have major impact, even when the lockfile is set to readonly, it can cause constraints to fail, at least in my experience. This wasn't a problem in earlier versions but since 1.1 of terraform they are more explicit in matching it appears.

Edited: this has been a very confusing week for me with terraform and automation, upgrading to terraform 1.1 with renovated repos has caused some issues and I've mixed up a few issues.

[rarkins]

Can you explain more? I didn't understand from this example.

[ekristen]

@rarkins apologies -- I edited by previous comment to clarify, I accidentally mixed a couple issues in my head after coming back from the weekend. Nevertheless I do believe that renovate having behavior that's different than terraform could cause problems, as it is with the zh hashes.

To be clear, I'm not hitting a constraint issue at this time (I thought I was originally) -- I am however hitting the issue that unless read-only is always specified because the zh hashes are missing the init will fail.

To give you an example on the constraint behavior between renovate and terraform ... when terraform updates the lock file would set the constraints to >= 2.0.0, >= 2.33.0, >= 3.56.0 but renovate will update it to ~> 3.56.0, which in this case happens to work, but I wonder if it could cause problems?

My main question would be why does terraform add the others if they aren't needed?

Here is a screenshot of a diff.

[viceice]

I don't see a issue on that constraints, as terraform adds a new one which forces a higher version, so the old one's are obsolete. They are combined with and, so it's the same to only require the highest range.

I also use read-only flag on my ci checks, so if any of renovates updates would cause a constraint issue, the checks will catch those.
I've never seen any issues since months.

[ekristen]

Generally speaking @viceice I would agree. I haven't seen an issue thus far, but at the same time, the behavior differs from the actual tool, this leads to the tools competing for who is right when updating dependencies. Renovate says X, but if you run a terraform init then it says Y.

[rarkins]

I think we should solve the zh hashes issue first and then see if the range list causes any problems in practice.

If it turns out it does, then we'd likely need to wait for the feature request to be implemented in terraform, which may be never.

[secustor]

Not adding zh hashes is the current Terraform behaviour if a cache/mirror is used. hashicorp/terraform#29958

IMO to move away from our custom implementation following issues have to at least be solved first:

• Add CLI command to upgrade lock file without full initialization hashicorp/terraform#27161 --> without this users give Renovate access to their TF state which contains secrets and other potential security risks
• Automatically cache zh and h1 hash values for all downloaded providers hashicorp/terraform#27811 --> without this, we would download every provider on every repo which translates for our fixtures to about 300 - 400 MB per run. 100 - 200 MB can be expected per renovated TF repo.

[ekristen]

@rarkins I like the approach and I agree, the range comments can be tabled, apologies for the confusion there.

@secustor I think the custom implementation is probably best given the scope of renovatebot and how it runs and allowing you all to cache hashes for subsequent runs is prudent.

Not adding zh hashes is the current Terraform behaviour if a cache/mirror is used. hashicorp/terraform#29958

I'll have to test this out a couple times to be sure, but I believe the assertion is partly correct. The zh hashes are added only on initial download of the provider files by terraform (ie fresh clone and terraform init).

For example if I just remove the lockfile and run terraform init, the zh hashes will not be added back, if I delete .terraform and the lockfile, then the zh hashes will be added.

Based on the 29958, if the files already exist in a local folder, then it's probably correct that the zh hashes won't be added. I'll have to test a network cache to see the behavior.

I do admit given this information that the behavior is then a bit inconsistent from the terraform side, however I still think adding the zh hashes is the correct thing to do.

[rarkins]

At the moment I'm leaning towards:

• if we can add the zh hashes without a lot of extra network traffic, it's definitely a good idea if someone can contribute it.
• if it means we need to download a lot of new files to do this and it can be significant traffic, we need to evaluate that

The fact that Terraform itself has known cases where it does not generate zh hashes either definitely helps the argument that Renovate's current approach is not "broken" at least.

[ekristen]

Honestly I'm a bit torn after our discussions now. Thanks for having them @rarkins and @secustor.

The fact that Terraform itself has known cases where it does not generate zh hashes either definitely helps the argument that Renovate's current approach is not "broken" at least.

I agree, this is actually the case when the file is retrieved from a non-registry location cache or mirror.

In fact, found this when reading the docs for the thousandth time.

zh this is a legacy hash format which is part of the Terraform provider registry protocol and is therefore used for providers that you install directly from an origin registry.

I guess the question is, how often are mirrors and caches being used to download the providers instead of the official registry?

I think that if it's trivial to add the zh it is worth it and based on what I'm reading in the Renovate code, all the files are already being downloaded unless you've cached it in the Renovate cache, so it's just a different format to generate and add?

However if it is not trivial or adds a lot of network traffic, I'm happy to close the issue.

Thank you again for talking through the issue. For now I've implemented all my automation to -lockfile=readonly to make this a non-issue for me at this time.

[z0rc]

I don't see a issue on that constraints, as terraform adds a new one which forces a higher version, so the old one's are obsolete. They are combined with and, so it's the same to only require the highest range.

This behavior is problematic, as it doesn't match what terraform does. constraints in lock file is a collection of all known constraints from current terraform code and modules. So when user works with modules, for example by adding or deleting a module to state, terraform will rewrite constraints in lock file. This results in unnecessary flapping between constraints generated by terraform vs generated by renovate.

[viceice]

Terraform won't add the zh hashes when the modules are already downloaded.

[z0rc]

@viceice my question was specifically about constraints, not hashes.