Panic while mirroring corrupted minio storage to openstack swift

[mishak87]

panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 37 [running]:
net/http.(*Request).write(0xc420691800, 0xb68940, 0xc42000e180, 0x0, 0x0, 0x0, 0xb6a9c0, 0xc420233360)
	/home/builder/.gimme/versions/go1.9.1.linux.amd64/src/net/http/request.go:620 +0x757
net/http.(*persistConn).writeLoop(0xc4202425a0)
	/home/builder/.gimme/versions/go1.9.1.linux.amd64/src/net/http/transport.go:1762 +0x201
created by net/http.(*Transport).dialConn
	/home/builder/.gimme/versions/go1.9.1.linux.amd64/src/net/http/transport.go:1187 +0xa53

mc version

• 2017-10-14T00:51:16Z
  Release-tag: RELEASE.2017-10-14T00-51-16Z
  Commit-id: 785e14a

System information

CentOS 7 Factory image on Openstack

[kannappanr]

Can you provide the entire log that is visible on the console?
Is this consistently reproducible?

[deekoder]

ping @mishak87 we would need a lot more info about the steps and the whole log to really diagnose the issue further.

[mishak87]

@deekoder Sorry I am on vacation. I will be back last week of the year.

[gnokix]

Hi there, I'm havig the same issue when mirroring two diferent storages one based on ceph and the other is s3.

the full command issued is:

mc --config-folder </pathto/config.json> mirror --quiet --overwrite --remove <swiftstorage/bucket> <s3storage/bucket>

a little experpt from the log file in which the issue ocurrs is:

cbe-000002-13/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-40/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-48/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-53/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-61/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-12/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-19/
`swift/cbe-000002-19/urn:oid:7561017` -> `s3/cbe-000002-19/urn:oid:7561017`
`swift/cbe-000002-19/urn:oid:7561069` -> `s3/cbe-000002-19/urn:oid:7561069`
panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 161 [running]:
net/http.(*Request).write(0xc4206f0900, 0xb73940, 0xc4207169c0, 0x0, 0x0, 0x0, 0xb75a00, 0xc420290d20)
        /opt/go/src/net/http/request.go:620 +0x757
net/http.(*persistConn).writeLoop(0xc420246c60)
        /opt/go/src/net/http/transport.go:1762 +0x201
created by net/http.(*Transport).dialConn
        /opt/go/src/net/http/transport.go:1187 +0xa53
cbe-000002-34/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-21/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-59/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-6/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-54/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-29/
Removing `s3/cbe-000002-29/urn:oid:7560993`.
`swift/cbe-000002-29/urn:oid:7561029` -> `s3/cbe-000002-29/urn:oid:7561029`
Removing `s3/cbe-000002-29/urn:oid:7560993`.
Total: 2.32 MB, Transferred: 2.32 MB, Speed: 79.12 KB/s
cbe-000002-31/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-58/
Total: 0 B, Transferred: 0 B, Speed: 0 B/s
cbe-000002-14/

[vadmeste]

There’re 3 types which aren’t comparable — maps, slices and functions. 

https://medium.com/golangspec/comparison-operators-in-go-910d9d788ec0

I think the above link can make us better understand what's going on.

[harshavardhana]

@deekoder Sorry I am on vacation. I will be back last week of the year.

@mishak87 can you provide us more information? we haven't been able to reproduce this issue at our end.

[harshavardhana]

@mishak87 can you provide us more information? we haven't been able to reproduce this issue at our end.

@mishak87 is there any information that you can provide us here?

[mishak87]

I will upgrade all minio servers in next two days and report back if issue still persists. Using latest mc client.

[mishak87]

Client does not panic anymore 👍 Upgrade to 2018-01-18T21:18:56Z of client and servers and running mc admin heal --debug --json ... worked.

[vadmeste]

I got this reported again, this most likely happens when mc received some unexpected data.

[fdaone]

I'm seeing this as well...

minio version on source host (Ubuntu Xenial):
~$ minio version
Version: 2018-01-02T23:07:00Z
Release-Tag: RELEASE.2018-01-02T23-07-00Z
Commit-ID: a6318db

~$ uname -a
Linux objectstore1 4.4.79-20170906-1335-7b4d138 #1 SMP Wed Sep 6 13:36:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

(I don't dare to upgrade this minio (distributed server setup) until I have managed to complete a full sync/mirror af all data)

mc version on target host (Ubuntu Bionic):
~# mc version
Version: 2018-12-12T19:02:22Z
Release-tag: RELEASE.2018-12-12T19-02-22Z
Commit-id: 69f967b

~# uname -a
Linux objectstore2 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Command:
~# mc mirror --watch --overwrite --remove objectstore1 objectstore2

Error output:
Actually got it twice within an hour (I'm in the process of mirroring 2 TB):

panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 15984841 [running]:
net/http.(*Request).write(0xc42182bc00, 0xcadd00, 0xc4215aed40, 0x0, 0x0, 0x0, 0xcaf120, 0xc42047bea0)
        /opt/go/src/net/http/request.go:624 +0x71b
net/http.(*persistConn).writeLoop(0xc420be79e0)
        /opt/go/src/net/http/transport.go:1825 +0x1ea
created by net/http.(*Transport).dialConn
        /opt/go/src/net/http/transport.go:1238 +0x97f

panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 5071 [running]:
net/http.(*Request).write(0xc42002ea00, 0xcadd00, 0xc420f964c0, 0x0, 0x0, 0x0, 0xcaf120, 0xc42045c1e0)
        /opt/go/src/net/http/request.go:624 +0x71b
net/http.(*persistConn).writeLoop(0xc42098aea0)
        /opt/go/src/net/http/transport.go:1825 +0x1ea
created by net/http.(*Transport).dialConn
        /opt/go/src/net/http/transport.go:1238 +0x97f

I have started it up again, this time with --debug to see if that will provide further info that might be useful.

[vadmeste]

@fdaone this is probaby a golang bug (golang/go#29768) but we still can add to simple workaround to avoid this situation.

[fdaone]

Thanks for the quick response. Adding a workaround in the mc code works for me. :-)

[lukekalish]

We are also dealing with the same issue. Happens several times a day and have implemented a service watcher to restart the mirror once it stops working. We're mirroring from one minio cluster to another. Any workaround to add stability would be great.

[fdaone]

This time using --debug (in hopes that reveals anything useful)

~# mc mirror --quiet --debug --watch --overwrite --remove objectstore1 objectstore2

[...]

mc: <DEBUG> GET /pdf-data/?marker=15%2F15%2F51%2F511515.pdf.gz&max-keys=1000 HTTP/1.1
Host: objectstore2:9000
User-Agent: Minio (linux; amd64) minio-go/v6.0.8 mc/2018-12-12T19:02:22Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXX/20190117/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190117T111350Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Thu, 17 Jan 2019 11:13:52 GMT
Server: Minio/RELEASE.2018-12-13T02-04-19Z (linux; amd64)
Vary: Origin
X-Amz-Request-Id: 157A9E712CB3DAC5
X-Minio-Deployment-Id: 2df698dc-aa6a-471d-a179-647e5a462e55
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  1.258065145s
    
`objectstore1/pdf/f/x/CFxkcTj.gz` -> `objectstore2/pdf/f/x/CFxkcTj.gz`
mc: <DEBUG> GET /pdf/?delimiter=%2F&max-keys=1000&prefix=f%2Fx%2FCFxkcTj.gz HTTP/1.1
Host: objectstore1:9000
User-Agent: Minio (linux; amd64) minio-go/v6.0.8 mc/2018-12-12T19:02:22Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXX/20190117/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190117T111352Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 17 Jan 2019 11:13:52 GMT
Server: Minio/RELEASE.2018-01-02T23-07-00Z (linux; amd64)
Vary: Origin
X-Amz-Request-Id: 157A9E71AB018B73

mc: <DEBUG> Response Time:  20.108248ms

mc: <DEBUG> HEAD /pdf/f/x/CFxkcTj.gz HTTP/1.1
Host: objectstore1:9000
User-Agent: Minio (linux; amd64) minio-go/v6.0.8 mc/2018-12-12T19:02:22Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXX/20190117/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190117T111352Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 60175
Accept-Ranges: bytes
Content-Type: application/octet-stream 
Date: Thu, 17 Jan 2019 11:13:52 GMT
Etag: "18ac1e7fd1de7f614fd2550b9feacc7b"
Last-Modified: Thu, 17 Jan 2019 11:13:52 GMT
Server: Minio/RELEASE.2018-01-02T23-07-00Z (linux; amd64)
Vary: Origin
X-Amz-Request-Id: 157A9E71AB451D8F

mc: <DEBUG> Response Time:  4.168958ms 

mc: <DEBUG> GET /pdf/f/x/CFxkcTj.gz HTTP/1.1
Host: objectstore1:9000
User-Agent: Minio (linux; amd64) minio-go/v6.0.8 mc/2018-12-12T19:02:22Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXX/20190117/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190117T111352Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 60175
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Thu, 17 Jan 2019 11:13:53 GMT
Etag: "18ac1e7fd1de7f614fd2550b9feacc7b"
Last-Modified: Thu, 17 Jan 2019 11:13:52 GMT
Server: Minio/RELEASE.2018-01-02T23-07-00Z (linux; amd64)
Vary: Origin
X-Amz-Request-Id: 157A9E71ABC90608

mc: <DEBUG> Response Time:  7.607506ms

mc: <DEBUG> GET /pdf/f/x/CFxkcTj.gz HTTP/1.1
Host: objectstore1:9000
User-Agent: Minio (linux; amd64) minio-go/v6.0.8 mc/2018-12-12T19:02:22Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXX/20190117/us-east-1/s3/aws4_request, SignedHeaders=host;if-match;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
If-Match: "18ac1e7fd1de7f614fd2550b9feacc7b"
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190117T111353Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 412 Precondition Failed
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 17 Jan 2019 11:13:53 GMT
Etag: "6e2ce1730a70800f9412dc023615b9ac"
Last-Modified: Thu, 17 Jan 2019 11:13:53 GMT
Server: Minio/RELEASE.2018-01-02T23-07-00Z (linux; amd64)
Vary: Origin
X-Amz-Request-Id: 157A9E71E662F99F

133
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key></Key><BucketName></BucketName><Resource>/pdf/f/x/CFxkcTj.gz</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>
0

mc: <DEBUG> Response Time:  982.97021ms

panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 35838 [running]:
net/http.(*Request).write(0xc420f76a00, 0xcadd00, 0xc4211e4180, 0x0, 0x0, 0x0, 0xcaf120, 0xc420ffc640)
        /opt/go/src/net/http/request.go:624 +0x71b
net/http.(*persistConn).writeLoop(0xc420a126c0)
        /opt/go/src/net/http/transport.go:1825 +0x1ea
created by net/http.(*Transport).dialConn
        /opt/go/src/net/http/transport.go:1238 +0x97f

[fdaone]

Also, I should probably add that I successfully mirrored everything a few days ago without stumbling on this error. At that time I wasn't mirroring from one minio setup to another, though. I was simply mirroring from a remote minio to a local filesystem (and I wasn't using --watch --overwrite --remove).

[fdaone]

Hi

I'm still seeing this with RELEASE.2019-01-24T01-40-23Z. Not as often as before, but it still makes the mc process die now and then which is a big pain point for me, since it takes mc mirror DAYS to "catch up" in the sync process (around 20 million small files in total). Fingers crossed that minio/minio#6494 will eventually be merged since having versioning in minio would make it somewhat redundant for me to mirror everything regularly to a secondary minio cluster which I then shut down and zfs snapshot for backup purposes.

root@objectstore2:~# mc mirror --quiet --json --overwrite --remove objectstore1/pdf objectstore2/pdf
[...]
{"status":"success","source":"objectstore1/pdf/P/Y/bNwqOkC.gz","target":"objectstore2/pdf/P/Y/bNwqOkC.gz","size":61555,"totalCount":24432,"totalSize":666135966}
{"key":"objectstore2/pdf/P/Y/PfBuBmN.gz","size":60203}
{"key":"objectstore2/pdf/P/Y/PfBuBmN.gz","size":0}
{"key":"objectstore2/pdf/P/Y/RJNfIZw.gz","size":59840}
{"key":"objectstore2/pdf/P/Y/TnARgsk.gz","size":60395}
{"key":"objectstore2/pdf/P/Y/RJNfIZw.gz","size":0}
{"key":"objectstore2/pdf/P/Y/TnARgsk.gz","size":0}
panic: runtime error: comparing uncomparable type minio.ErrorResponse

goroutine 284863 [running]:
net/http.(*Request).write(0xc0008a0200, 0xd39280, 0xc000b62440, 0x0, 0x0, 0x0, 0xd3a6a0, 0xc00039abe0)
        /opt/go/src/net/http/request.go:647 +0x74c
net/http.(*persistConn).writeLoop(0xc000a3a120)
        /opt/go/src/net/http/transport.go:1888 +0x1b8
created by net/http.(*Transport).dialConn
        /opt/go/src/net/http/transport.go:1339 +0x966

root@objectstore2:~# mc --version
mc version RELEASE.2019-01-24T01-40-23Z

[harshavardhana]

Assigning this to @vadmeste due the original fix here minio/minio-go#1066

[vadmeste]

@fdaone I've just checked, version RELEASE.2019-01-24T01-40-23Z doesn't contain the fix (it doesn't contain the updated minio-go library), Headers here is causing the issue https://github.com/minio/mc/blob/RELEASE.2019-01-24T01-40-23Z/vendor/github.com/minio/minio-go/api-error-response.go#L56

Please feel free to reopen this issue if you have further questions or you are still seeing the error.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.