[improvement] Document or provide usage for how to re-run/backfill provenance generation on failure

[asraa]

per some conversation with @droot on using the workflows

Describe the bug
Right now, if a project performs a release that generates SLSA provenance, and the provenance generation step fails, then they not have a GitHub release at a tag that does not contain the provenance. They may need to update the builder or change the workflow. They also prefer immutable tags and not force pushing tags after the fix. There are some alternatives and things we can suggest:

1. [Preventative] Use a scheduled cron job to continuously dry-run the release or use on push to verify the release.
2. [Workarounds] Use a workflow_dispatch trigger to re-upload the provenance, assuming the release build is reproducible. This will not replicate the tag information because of the workflow fix that required a commit push.
3. More?

Either way, we should add some documentation to the README.md for suggested usage. Let's definitely suggest the preventative approach.

[asraa]

Another request here about backfilling SLSA provenance:
grpc-ecosystem/grpc-gateway#2987 (comment)

The most simple way would only be if the build is reproducible and is re-executed in the workflow at a certain tag which can be given as a tag parameter of the caller workflow.

We could even expedite the upload asset part with #713

[droot]

• [Workarounds] Use a workflow_dispatch trigger to re-upload the provenance, assuming the release build is reproducible. This will not replicate the tag information because of the workflow fix that required a commit push.

Can you share a pointer to doc or steps describing - how this can be done in Github. In case of our project kpt, the builds are reproducible.

[asraa]

In case of our project kpt, the builds are reproducible.

Working on it now with an example project!

[asraa]

@johanbrandhorst @droot I was working on a re-trigger for grpc-gateways workflow.

• Added a workflow_dispatch that tags a tag_name as parameter
• Re-runs goreleaser with the checkout at the tag (on re-run, --skip-publish)
• Only upload-assets for provenance when this is a push event: otherwise, it won't know what release to upload to. We are working on a configurable release URL: feat: upload provenance to release via a tag-name #713

However, when trying with grpc-gateway, I didn't get reproducible builds.

This is the workflow that I've been running on my own fork https://github.com/asraa/grpc-gateway/blob/master/.github/workflows/release.yml. @droot I can also take a look at yours.

[johanbrandhorst]

That's fair enough, I can't say why we don't have reproducible builds, the build timestamp might be making it into the binary without some extra build arguments. Feel free to just make it easy to dry-run test the workflow without trying to recreate previous releases.

[asraa]

Feel free to just make it easy to dry-run test the workflow without trying to recreate previous releases.

Sounds good, expect a PR on your end soon!