Version v4.27.1 removed from GitHub Repo

[therealdwright]

Terraform Version

Terraform v1.1.7
on darwin_arm64

Your version of Terraform is out of date! The latest version
is 1.2.5. You can update by downloading from https://www.terraform.io/downloads.htm

Affected Resource(s)

All GitHub Resources

Output

╷
│ Error: Failed to install provider
│ 
│ Error while installing integrations/github v4.27.1: could not query provider registry for
│ registry.terraform.io/integrations/github: failed to retrieve authentication checksums for provider:
│ 404 Not Found returned from github.com

Expected Behavior

Init completes successfully.

Actual Behavior

Init fails with above output.

Steps to Reproduce

terraform init -upgrade with below provider config:

terraform {
  required_version = ">= 1.0"
  required_providers {
    github = {
      source  = "integrations/github"
      version = "~> 4.27"
    }
  }
}

v4.27.1 looks to be removed from the github repo tags whilst still being listed on the provider:
https://registry.terraform.io/providers/integrations/github/4.27.1

[kfcampbell]

I'm sorry, I messed up and accidentally merged a binary into the source tree! I removed the tags for v4.27.0 and v4.27.1 as those were the affected releases. I'm seeing if I can get the registry to take down those versions as well. Can you use v4.28.0 in the meantime? The content is the same as v4.27.1.

[stvngrsh]

Lol was waiting for someone to post this. Was sitting around waiting for the github action to finish the 4.28.0 release. Runs are succeeding now assuming that OP just has something like > 4.0 set in his provider file.

[therealdwright]

Lol was waiting for someone to post this. Was sitting around waiting for the github action to finish the 4.28.0 release. Runs are succeeding now assuming that OP just has something like > 4.0 set in his provider file.

We were relying on features released in v4.27.0 hence the specific provider config posted above.

[therealdwright]

Can confirm working with the provider config of:

terraform {
  required_version = ">= 1.0"
  required_providers {
    github = {
      source  = "integrations/github"
      version = "~> 4.28"
    }
  }
}

[ghost]

It can be fixed by an update to latest version but I don't know why 4.27.1 is removed so fast...

[jblazek]

Can 4.27.1 be restored? This is breaking existing pipelines for us.

[LLBlumire]

Seconding @jblazek , this broke a load of nix pipelines for me

[kfcampbell]

Explanation of the binary incident

TL;DR: I messed up and merged a binary into the project's source tree that increased the size of the repo dramatically (20.01MiB to 32.00MiB). In order to fix this and ensure that future repository clones don't include this bloated size, I force-pushed the repository to the commit immediately before the binary was introduced and removed the tags for the versions with the binary included. I then replayed the two useful commits afterward and released a new version. Sorry for the inconvenience!

What happened

This PR introduced a helpful change in the GitHub repository data source to allow pulling only protected branches. However, it mistakenly included a binary named terraform-provider-github that should not have been introduced to the source tree. I meant to remove this binary before merging, and solely due to my own mistakes, I merged the PR in 13 days ago and created a release.

Afterward, I panicked and reverted the commit in this PR, without thinking clearly: of course a revert commit would still keep the binary in the tree and affect the overall clone size.

Shortly after, I also merged this PR to revert a previous PR which broke existing behavior and also created a release for that.

Now the state of the provider's source tree was awkward: a PR had been merged containing an unnecessary binary that bloated the repo clone size, and there was a PR on top of it that reverted a bug that needed to be kept. Furthermore, two releases had been created with the binary included.

Size comparisons

Cloning the repo with the binary included required 32 MiB:

sh$ git clone [https://github.com/integrations/terraform-provider-github](https://github.com/integrations/terraform-provider-github)
Cloning into 'terraform-provider-github'...
remote: Enumerating objects: 21444, done.
remote: Counting objects: 100% (498/498), done.
remote: Compressing objects: 100% (281/281), done.
remote: Total 21444 (delta 278), reused 398 (delta 209), pack-reused 20946
Receiving objects: 100% (21444/21444), 32.00 MiB | 11.12 MiB/s, done.
Resolving deltas: 100% (11637/11637), done.

Without the binary added, the cloning process requires just 20.01 MiB:

$ git clone https://github.com/integrations/terraform-provider-github
Cloning into 'terraform-provider-github'...
remote: Enumerating objects: 21441, done.
remote: Counting objects: 100% (495/495), done.
remote: Compressing objects: 100% (278/278), done.
remote: Total 21441 (delta 277), reused 394 (delta 209), pack-reused 20946
Receiving objects: 100% (21441/21441), 20.01 MiB | 510.00 KiB/s, done.
Resolving deltas: 100% (11623/11623), done.

. How can a binary that according to the original PR takes 24.5 MB of space make up a difference of 12 MiB in the repo?

According to this git.kernel.org post, the compression algorithm git uses is zlib, which can be accessed (among other ways) using the CLI zlib-flate.

Running zlib-flate on the binary can show us it achieves close to 50% compression:

 sh$ ls -l terraform-provider-github
-rwxrwxr-x 1 kfcampbell kfcampbell 25M Jul 28 12:50 terraform-provider-github*

 sh$ zlib-flate -compress < terraform-provider-github > terraform-provider-github.zip

 sh$ ls -l terraform-provider-github.zip
-rw-rw-r-- 1 kfcampbell kfcampbell 13M Jul 28 12:55 terraform-provider-github.zip

which explains the 12 MB difference we're seeing.

Resolution

I decided to remove the binary from the source tree. To do this, I initally used git filter-repo to rewrite history without the binary. However, doing so led to a completely different history: my fork was listed as being "1003 commits behind, 1002 commits ahead" of origin. That would've been really disruptive to all open PRs and forks.

After discussing with a more knowledgeable colleague, I instead decided to force-push the repository to this commit, the last one before the binary was introduced. Afterwards, I replayed the changes that were added in PR 1162 here, and re-reverted 754d73f here.

I then removed the tags v4.27.0 and v4.27.1 (git push --delete origin 4.27.0 and git push --delete origin 4.27.1) so that the binary could not be referenced by any commit or tag.

Afterwards, cloning the repository is back to its normal size:

 sh$ git clone https://github.com/integrations/terraform-provider-github
Cloning into 'terraform-provider-github'...
remote: Enumerating objects: 21441, done.
remote: Counting objects: 100% (495/495), done.
remote: Compressing objects: 100% (278/278), done.
remote: Total 21441 (delta 277), reused 394 (delta 209), pack-reused 20946
Receiving objects: 100% (21441/21441), 20.01 MiB | 13.06 MiB/s, done.
Resolving deltas: 100% (11623/11623), done.

Unfortunately, as a consequence of deleting the tags 4.27.0 and 4.27.1, the Hashicorp registry versions of those plugins fail to download and cannot be used. I did not realize this would happen and I apologize for the hassles caused by it. Those affected should use v4.28.0, which is the same code as 4.27.1 minus the spurious binary. I've reached out to Hashicorp about removing those releases from the registry and have not heard back yet.

What actions are required of open PRs/Forks?

Open PRs may simply merge the main branch in order to avoid any changes in their diffs.

Forks should be updated to pull from the latest main branch.

What would I do differently next time?

So much! I've learned a lot during this episode, and not just about git filter-branch.

• Include more guardrails so that binaries won't be added to the tree. This PR is a first attempt at doing so.
• Reconsider force-pushing at all: is 12MiB tolerable to leave in the source tree in order to avoid rewriting history? Perhaps.
• Reach out to Hashicorp before removing the tags for releases v4.27.0 and v4.27.1. Perhaps they have a procedure for circumstances of this nature and/or could've provided helpful advice and historical context.
• Make extra-extra sure there are no files without purpose lingering in PRs to be merged.

Conclusion

I'm sorry for all the hassle! It won't happen again.

[therealdwright]

Thanks for the comprehensive write-up @kfcampbell - things happen and it's OK - I appreciate you taking the time to explain what happened.

[dvargas92495]

I'm using cdktf and it's still downloading this version of the GitHub Provider. I tried editing the cdktf.json to include:

"terraformProviders": [
    { "name": "github", "source": "integrations/github", "version": "4.28.0" }
],

to no avail, my plans still download 4.27.1 from the registry. Is there an obvious step I'm missing?

EDIT:

I updated my @cdktf/provider-github npm library and now I'm seeing that version 4.28.0 is not available:

EDIT 2:

I have realized that this is user error, from one of my submodules requiring 4.2.0 at the same time as 4.28.0. Carry on 🤦

[eumoh1601]

Thankyou for the write-up @kfcampbell it's OK - I appreciate that you took the time to explain this - the version 4.28.0 worked for me.