AWS S3 endpoint configuration not working

[Pacobart]

We are using thanos as a sidecar to prometheus to ship metrics data to an AWS S3 bucket. We are getting errors because thanos is using the s3.dualstack..amazonaws.com endpoint. In the objstore secret passed into thanos we are applying the endpoint parameter but it isn't actually being used.

objstore:

type: s3
config:
  bucket: mybucket
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1 
  signature_version2: false
  put_user_metadata: {"X-Amz-Acl": "bucket-owner-full-control"}
  sse_config:
    type: "SSE-KMS"
    kms_key_id: XXXXX

Thanos is running as a container on kubernetes. We are using the bitnami/thanos::0.29.0-scratch-r0 image.
Thanos, Prometheus and Golang version used:

Object Storage Provider:
AWS S3
What happened:
Thanos is trying to upload data to mybucket.s3.dualstack.us-east-1.amazonaws.com
What you expected to happen:
I expect it to use the endpoint provided in the configuration and to use mybucket.s3.us-east-1.amazonaws.com
How to reproduce it (as minimally and precisely as possible):
We deploy the kube-prometheus-stack, enable thanos and add the objstore config.
Full logs to relevant components:
│ level=warn ts=2023-10-13T22:12:21.474412821Z caller=sidecar.go:346 err="check exists: stat s3 object: Head "https://mybucket.s3.dualstack.us-east-1.amazonaws.com/01HCK03E9DZ0PD0MKJCGNSEVF9/meta.json\": net/http: TLS handshake timeout" uploaded=0

Anything else we need to know:

[jnyi]

which version are you using, we encountered similar problem for the latest one: #6785

[Pacobart]

@jnyi We are using version 0.29.0

[Pacobart]

minio/minio-go#1055
Found this is a limitation of how the mino package. They have hardcoded dns names which is a bad practice. After I updated the endpoint to use the regex matched vpc endpoint I was able to bypass this issue.