Support S3 endpoints for module sources

[alexjeen]

Terraform Version

Terraform v1.7.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.41.0

Use Cases

Because bucket policies are limited in size (4kb) you must use bucket access points to give access if you reach the limit, every bucket access point can have it's own bucket policy.

We have a module S3 bucket, and it has modules for a lot of customers in a lot of different AWS accounts so we need to give access to all these customer accounts.

Attempted Solutions

Currently we are solving it by using the s3 URL directly:

terraform {
  source = "s3::https://bucketname.s3.eu-west-1.amazonaws.com/module.zip"
}

However when we try to use a bucket endpoint:

terraform {
  source = "s3::https://endpointname-12391291991.s3-accesspoint.eu-west-1.amazonaws.com/module.zip"
}

We get an error:

ERRO[0000] downloading source url s3::https://endpointname-12391291991.s3-accesspoint.eu-west-1.amazonaws.com/module.zip
1 error occurred:
        * NoSuchBucket: The specified bucket does not exist
        status code: 404, request id: MTA7QQC63CYFPV61, host id: /oB9iw8e0zLMr4x0Vz2sCW49NLpQ9DC4w8bG4M52iSMXiGtyyipwcSkNJWVy8jpZO6ZtgDv+160=

Proposal

Change this function: https://github.com/hashicorp/terraform/blob/main/internal/getmodules/moduleaddrs/detect_s3.go
To support access points (the url is different), the authentication is the same.

And access point URL looks like this:

https://accesspointname-123456789012.s3-accesspoint.eu-west-1.amazonaws.com/module.zip

Where 123456789012 is the account id of the access point.

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!