Restic doesn't use S3 Dual-Stack endpoints, breaking IPv6 support

[andreaso]

Output of restic version

$ /opt/restic/0.9.3/restic version
restic 0.9.3 compiled with go1.11.1 on linux/amd64
$

How did you run restic exactly?

This being on an host with only IPv6 connectivity.

The S3 bucket arrakis-v6lab-restic have been pre-created within the eu-central-1 region.

The following environment variables have been defined for AWS access.

$ export | grep AWS
declare -x AWS_ACCESS_KEY_ID="REDACTED"
declare -x AWS_DEFAULT_REGION="eu-central-1"
declare -x AWS_SECRET_ACCESS_KEY="REDACTED"
$

Doing the initial init attempt, based on example in documentation.

$ /opt/restic/0.9.3/restic --password-file ~/.secret/restic_password --repo s3:s3.amazonaws.com/arrakis-v6lab-restic init
Fatal: create repository at s3:s3.amazonaws.com/arrakis-v6lab-restic failed: client.BucketExists: Get https://s3.amazonaws.com/arrakis-v6lab-restic/?location=: dial tcp 52.216.164.125:443: connect: network is unreachable

$

Followed up by explicitly providing a dual-stack endpoint.

$ /opt/restic/0.9.3/restic --password-file ~/.secret/restic_password --repo s3:s3.dualstack.eu-central-1.amazonaws.com/arrakis-v6lab-restic init
Fatal: create repository at s3:s3.dualstack.eu-central-1.amazonaws.com/arrakis-v6lab-restic failed: client.BucketExists: Head https://arrakis-v6lab-restic.s3-eu-central-1.amazonaws.com/: dial tcp 52.219.73.93:443: connect: network is unreachable

$

Manually providing an IPv6 address for arrakis-v6lab-restic.s3-eu-central-1.amazonaws.com did the trick.

$ host s3.dualstack.eu-central-1.amazonaws.com
s3.dualstack.eu-central-1.amazonaws.com has address 52.219.74.52
s3.dualstack.eu-central-1.amazonaws.com has IPv6 address 2a05:d050:4080:80:34db:4a31::
$

$ grep arrakis-v6lab-restic /etc/hosts
2a05:d050:4080:80:34db:4a31::    arrakis-v6lab-restic.s3-eu-central-1.amazonaws.com
$

$ /opt/restic/0.9.3/restic --password-file ~/.secret/restic_password --repo s3:s3.dualstack.eu-central-1.amazonaws.com/arrakis-v6lab-restic init
created restic repository 7601f5be4a at s3:s3.dualstack.eu-central-1.amazonaws.com/arrakis-v6lab-restic

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
$

What backend/server/service did you use to store the repository?

AWS S3

Expected behavior

That restic would either use the S3 dual-stack endpoints by default, or that there would be a way to tell restic to do so.

Actual behavior

Restic uses IPv4-only S3 endpoints, unless "lied" to it.

Steps to reproduce the behavior

1. Ensure an IPv6-only environment
2. Follow the steps above.

Do you have any idea what may have caused this?

After having used the initially provided S3 endpoint to lookup the bucket location restic/minio-go appear to construct a new IPv4-only S3 endpoint based on github.com/minio/minio-go/s3-endpoints.go.

Confirmed by the fact that the following monkey patching solved my immediate problem.

$ git diff
diff --git a/vendor/github.com/minio/minio-go/s3-endpoints.go b/vendor/github.com/minio/minio-go/s3-endpoints.go
index 05892950..adc400e8 100644
--- a/vendor/github.com/minio/minio-go/s3-endpoints.go
+++ b/vendor/github.com/minio/minio-go/s3-endpoints.go
@@ -27,7 +27,7 @@ var awsS3EndpointMap = map[string]string{
        "eu-west-1":      "s3-eu-west-1.amazonaws.com",
        "eu-west-2":      "s3-eu-west-2.amazonaws.com",
        "eu-west-3":      "s3-eu-west-3.amazonaws.com",
-       "eu-central-1":   "s3-eu-central-1.amazonaws.com",
+       "eu-central-1":   "s3.dualstack.eu-central-1.amazonaws.com",
        "ap-south-1":     "s3-ap-south-1.amazonaws.com",
        "ap-southeast-1": "s3-ap-southeast-1.amazonaws.com",
        "ap-southeast-2": "s3-ap-southeast-2.amazonaws.com",
$

Do you have an idea how to solve the issue?

• Work with the minio-go library so that it either provides dual-stack endpoints by default, or provides an option to do so.

and/or.

• When the user provides a sufficient specific endpoint, continue using it.

Did restic help you or made you happy in any way?

• I love that restic allows me to perform smart backups against a dumb storage backend.
• I'm kind of happy that this bug gives me one more nudge to start learning Go.

[fd0]

Thanks for reporting this issue. It's indeed rooted in the minio-go library we're using for accessing s3. As far as I know it's necessary to contact s3 and then use the endpoint provided by s3 afterwards (for region configuration maybe?).

Ping @harshavardhana, is it a good idea to always use the dualstack endpoints for s3 by default in minio-go? Here's the relevant documentation that I found: https://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html

[harshavardhana]

Yes we can do that is dualstack endpoints always available ?

[andreaso]

Based on https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region S3 dualstack endpoints appear to be available in all non-China regions.

@harshavardhana: Or did you mean/wonder something else?

[harshavardhana]

Okay then we can simply default to those endpoints instead and never have to change it.

[fd0]

Cool! 👍

[andreaso]

Anything I can/should help out with?

[harshavardhana]

Anything I can/should help out with?

Feel free to send a PR @andreaso to github.com/minio/minio-go

[andreaso]

Ok, will be doing that next year then :-) Feel I need to familiarize myself with the library a bit first, so that I'll be able to properly test my changes.

[fd0]

Just a tip: don't over-complicate this, just make the change and test it with restic :)

[andreaso]

So, minio/minio-go#1055 at least partially improves the situation. You still have to explicitly provide the right dualstack endpoint to the --repo flag, but at least the follow-up connections work without the need for any /etc/hosts trickery.

minio/minio-go#1060 is also somewhat relevant to Restic, as it provides support for the new eu-north-1 region.

These changes are available as of https://github.com/minio/minio-go/releases/tag/v6.0.12.

I guess the next step would be a PR bumping Restic's minio-go version? On that note, what's the deal with the "incompatibility" of the current v6.0.7+incompatible version?

[fd0]

I'll update the version, the +incompatible suffix means that minio-go is not yet using "Go Modules", the new dependency management stuff recently introduced.

[harshavardhana]

I'll update the version, the +incompatible suffix means that minio-go is not yet using "Go Modules", the new dependency management stuff recently introduced.

That's fixed as well @fd0

[fd0]

Hm, how so? I can see that there's now a go.mod, but you're not doing "semantic import versioning": the module name (and import path!) should be github.com/minio/minio-go/v6. This video tutorial is very helpful and outlines many pitfalls: https://www.youtube.com/watch?v=ms5l0zxC-uM

[harshavardhana]

Isn't you who should be doing that import ? @fd0

[fd0]

Yeah, every consumer of minio-go must do that when they use Go with modules enabled. But as far as I know the correct import path (including the version if > 1) must also be included in the module file itself. I think the video explains it in great detail...

[fd0]

@harshavardhana I'm getting errors in our CI tests with AWS:

    suite.go:167: client.BucketExists: Head https://[secure].s3.dualstack.us-east-1.amazonaws.com/: 301 response missing Location header

Is this a known issue?

[harshavardhana]

In this situation you haven't provided the right region

• either GetBucketLocation API is disabled perhaps in your IAM policies for us to get the region directly.
• region should be statically provided for the concerned bucket

Can you check, will validate on our end as well .

[fd0]

Hm, I'll have a look. It only fails occasionally, and only for Go 1.11. Our CI tests run with Go 1.9, 1.10 and 1.11, the former two pass the test...

[andreaso]

Doesn't look like S3 is tested with those older Go versions.

RESTIC_TEST_CLOUD_BACKENDS=0

[harshavardhana]

Oh I think the problem is perhaps in Redirect code which we have, perhaps S3 sends 301 without Location header and Go http is returning an error..

Can you provide a http trace of this situation ? using TraceOn etc?

[fd0]

Oh wow, I forgot, thanks for the reminder @andreaso!

I'll try to reproduce it locally, with a trace.

[fd0]

Indeed, the credentials used for testing are not allowed to query the location, hm:

$ go test -v ./internal/backend/s3 -run S3
---------START-HTTP---------
GET /restic-test-travis/?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/v6.0.14
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20190211/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20190211T184959Z
Accept-Encoding: gzip

HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 11 Feb 2019 18:49:59 GMT
Server: AmazonS3
X-Amz-Id-2: N9SoMrEpq6ru+xZUI3Z4dxlRiihdeZsZZSZm1BDFiKZOOMPM5ho+AWKqzsgRsLORq8oITaNMYBA=
X-Amz-Request-Id: 8CBA22298F5B3C59

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8CBA22298F5B3C59</RequestId><HostId>N9SoMrEpq6ru+xZUI3Z4dxlRiihdeZsZZSZm1BDFiKZOOMPM5ho+AWKqzsgRsLORq8oITaNMYBA=</HostId></Error>
0
---------END-HTTP---------
--- FAIL: TestBackendS3 (2.61s)
    s3_test.go:308: run tests
    suite.go:167: client.BucketExists: Head https://restic-test-travis.s3.dualstack.us-east-1.amazonaws.com/: 301 response missing Location header
FAIL
FAIL	github.com/restic/restic/internal/backend/s3	2.618s

This worked before though, this was the IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1494536175000",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::restic-test-travis/*"
            ]
        },
        {
            "Sid": "Stmt1494536236000",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::restic-test-travis"
            ]
        }
    ]
}

I've now added s3:GetBucketLocation to the second list and now it works:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1494536175000",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::restic-test-travis/*"
            ]
        },
        {
            "Sid": "Stmt1494536236000",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::restic-test-travis"
            ]
        }
    ]
}

Is this something that minio-go just requires? I noticed that we also have it in the list of permissions in the tutorial here: https://restic.readthedocs.io/en/latest/080_examples.html

[fd0]

I just noticed: the error message is not helpful at all, and the last response also is not a 301, but a 403. Hm. At least the error message could be better :)

[harshavardhana]

I just noticed: the error message is not helpful at all, and the last response also is not a 301, but a 403. Hm. At least the error message could be better :)

Well at that situation minio-go automatically falls back to us-east-1 and then allows S3 to send a redirect, if we see a location header we redirect there too @fd0 - so in a way lot of things failed :-)

Looks like for HEAD bucket S3 doesn't set a Location header but sets 301 nonetheless. Sort of incorrect HTTP standard implementation. HEAD doesn't have a body so Location header is the only way to redirect.

Is this something that minio-go just requires? I noticed that we also have it in the list of permissions in the tutorial here: https://restic.readthedocs.io/en/latest/080_examples.html

S3 has many regions and people have buckets in many different regions - it is cumbersome for users to keep typing this or statically configure it which allows them to not use other buckets in different regions. This is what AWS CLI or AWS SDKs lack. So minio-go uses GetBucketLocation() to fetch the region and construct the right URL based on the bucket.

So yes you need those permissions.

[fd0]

Using the HTTP request debugging built into restic we can get a bit more information:

First, it tries to find out the location, response is 403:

$ DEBUG_FILES=debug/*.go go test -tags debug -v ./internal/backend/s3 -run S3

debug enabled
=== RUN   TestBackendS3
debug/round_tripper_debug.go:77	debug.loggingRoundTripper.RoundTrip	20	------------  HTTP REQUEST -----------
GET /restic-test-travis/?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/v6.0.14
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJSQEUXZSP56YBL6Q/20190211/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9f1b2e0a258b9c0eaa4ed24a58eb2dcd40931f2d65a206c7c6c4202312578b5b
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20190211T190842Z
Accept-Encoding: gzip


debug/round_tripper_debug.go:90	debug.loggingRoundTripper.RoundTrip	20	------------  HTTP RESPONSE ----------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 11 Feb 2019 19:08:42 GMT
Server: AmazonS3
X-Amz-Id-2: Z/m5sFAHivYY5w8vc/icZAc0XoN7/DScaXdc+7aac29+2eSl5M3VzYDXoq217MitKba19US9+vU=
X-Amz-Request-Id: A3398AF745BCC5F3

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>A3398AF745BCC5F3</RequestId><HostId>Z/m5sFAHivYY5w8vc/icZAc0XoN7/DScaXdc+7aac29+2eSl5M3VzYDXoq217MitKba19US9+vU=</HostId></Error>
0

Next, it tries listing the files, and gets a 301 response without a Location header, but the endpoint restic-test-travis.s3.dualstack.eu-central-1.amazonaws.com is in the response XML:

debug/round_tripper_debug.go:77	debug.loggingRoundTripper.RoundTrip	20	------------  HTTP REQUEST -----------
GET /?delimiter=%2F&max-keys=1000&prefix=test-1549912122128613913%2Fkeys%2F HTTP/1.1
Host: restic-test-travis.s3.dualstack.us-east-1.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/v6.0.14
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJSQEUXZSP56YBL6Q/20190211/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=025a102a07876af7e544b716988db57592c0e562365a6ca64b2ff01bcb0ea15a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190211T190842Z
Accept-Encoding: gzip


debug/round_tripper_debug.go:90	debug.loggingRoundTripper.RoundTrip	20	------------  HTTP RESPONSE ----------
HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 11 Feb 2019 19:08:43 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: 9jEi8t+H3Tdm5u3q9hMUVbul/5kiv/6Ndt3uueeQbO9gp/12ZHd2iu/TSK6LMi//D25w9K4qfC8=
X-Amz-Request-Id: A06FCBD6F31830EF

1e6
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>restic-test-travis.s3.dualstack.eu-central-1.amazonaws.com</Endpoint><Bucket>restic-test-travis</Bucket><RequestId>A06FCBD6F31830EF</RequestId><HostId>9jEi8t+H3Tdm5u3q9hMUVbul/5kiv/6Ndt3uueeQbO9gp/12ZHd2iu/TSK6LMi//D25w9K4qfC8=</HostId></Error>
0

[fd0]

Very odd that the service returns 301 without Location, hm.

[basilmusa]

When minio-go version got upgraded to 6.0.14, NetApp S3 Object Storage is giving problems.
See: #2273