MC cp size is bigger than real size #2297

kybom · 2017-10-26T02:08:39Z

Expected behaviour

MC cp size is equal to real zise.

Actual behaviour

mc cp size(1.6G) is bigger than real zise(1.59G).

1.60 GB / 1.59 GB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.52% 412.63 KB/s 1h7m45s

Steps to reproduce the behaviour

mc version

(paste output of mc version)
Version: DEVELOPMENT.GOGET
Release-tag: DEVELOPMENT.GOGET
Commit-id: DEVELOPMENT.GOGE

System information

centos 6.5

The text was updated successfully, but these errors were encountered:

vadmeste · 2017-10-26T10:38:01Z

I think this happens when mc retries a failed request in the middle of the operation.

Thanks for reporting it @kybom

ebozduman · 2017-10-31T01:25:54Z

@kybom ,
Could you please tell us what exact command you've used when you hit this issue and was the copy done on a single file?
I am assuming you've done a simple copy on a single file like;

$ mc cp /tmp/testfile /var/tmp/testfile_copy

but I am having hard time reproducing it.

ebozduman · 2017-11-02T22:10:48Z

@kybom ,
I spent quite some time on reproducing this issue by initiating network disruption (turning wifi off/on) on the server side, while mc cp was going on on the client side. When the network came back on, the hung copy process started continuing and hence my repro attempt failed to reproduce the issue. That is, the copy process completed without any problem and displayed the sizes correctly.
I did repeat this test with a single 1.59GB file and with multiple files (100MB, 150MB, and 200MB) without any issues.
Code review also did not reveal any obvious/possible bugs in the related logic.

So, at this point, we need your help on reproducing the issue.
Please provide us more details on the reproduction; whether this issue is still reproducible/repeatable on your system/setup, what the exact size of the file in bytes is, how many files is involved in the copy process. Basically we need any information that is relevant or might be relevant to this issue, for us to keep looking into it.
Thanks...

kybom · 2017-11-04T09:21:24Z

@ebozduman This issue is still reproducible on my system. The file size is 1708801009 Bytes. One file.

kybom · 2017-11-04T09:24:46Z

Just a copy mc cp kk_20171021.tar.gz minio/mybucket/

harshavardhana · 2017-11-04T17:19:36Z

Which version of mc are you using ?. Can you use the latest and let us know if it's reproducible ?

kybom · 2017-11-04T19:09:28Z

how to solve the issue

mc: Failed to copy xxxxx_20171021.tar.gz. Post https://www.xxxxxx.net:9000/xxxxx/xxxxx_20171021.tar.gz?uploads=: net/http: invalid header field value "unconfined_u:object_r:user_home_t:s0\x00" for key X-Amz-Meta-Security.selinux

harshavardhana · 2017-11-04T20:53:58Z

This is expected the object storage you are using uses incompatible http header values on objects. This is part of Go http client we do not have control on this. So we cannot fix it.

The object storage you are using is implemented incorrectly in following HTTP spec.

https://github.com/golang/net/blob/master/lex/httplex/httplex.go#L270

You should ask your object storage vendor to fix this properly and follow http spec.

harshavardhana · 2017-11-04T20:55:31Z

Btw is your server by any chance minio ?

harshavardhana · 2017-11-04T21:09:19Z

const Async = require('async')
const AWS = require('aws-sdk')
const UUID = require('node-uuid')

var options = {
    accessKey: 'USWUXHGYZQYFYFFIT3RE',
    secretKey: 'MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03',
    region: 'us-east-1',
    endpoint: 'http://localhost:9000'
};

let s3opts = {
    signatureVersion: 'v4',
    accessKeyId: options.accessKey,
    secretAccessKey: options.secretKey,
    region: options.region,
}

s3opts = Object.assign(s3opts, {
    endpoint: options.endpoint,
    s3ForcePathStyle: true,
    s3BucketEndpoint: false,
    sslEnabled: false,
})

var s3Client = new AWS.S3(s3opts);
var bucket = "testbucket";

s3Client.putObject({
    Bucket: bucket,
    Key: "my-objectname",
    Metadata: {"security.selinux": "unconfined_u:object_r:user_home_t:s0\x00"},
}, (err, data) => {
    if (err) {
        console.log(err);
    }
    console.log(data);
})

This example throws an error

{ NetworkingError: The header content contains invalid characters
    at ClientRequest.setHeader (_http_outgoing.js:370:11)
    at new ClientRequest (_http_client.js:131:14)
    at Object.request (http.js:26:10)
    at features.constructor.handleRequest (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/http/node.js:43:23)
    at executeSend (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/event_listeners.js:241:29)
    at Request.SEND (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/event_listeners.js:256:9)
    at Request.callListeners (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/sequential_executor.js:101:18)
    at Request.emit (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/request.js:668:14)
    at Request.transition (/home/harsha/aws-sdk/js/node_modules/aws-sdk/lib/request.js:22:10)
  message: 'The header content contains invalid characters',
  code: 'NetworkingError',
  region: 'us-east-1',
  hostname: 'localhost',
  retryable: true,
  time: 2017-11-04T21:06:53.361Z }

This is the expected error i would see on AWS SDKs, Minio SDKs - i am not sure how you created an object with the header value which is incorrect according to HTTP spec.

kybom · 2017-11-06T06:15:58Z

The release mc version does not the net/HTTP error. The lastest commit has the error"net/http...".
Minio server is mine, "VERSION:
2017-09-29T19:16:56Z".

harshavardhana · 2017-11-06T19:33:27Z

The release mc version does not the net/HTTP error. The lastest commit has the error"net/http...".
Minio server is mine, "VERSION:
2017-09-29T19:16:56Z".

I do not follow anything that is being said here can you open a new issue and close this one? @kybom

ebozduman · 2017-11-06T20:39:40Z

Hey @kybom ,

Sorry, but we are kind of confused here.

Please clarify the followings:

You still see the original size mismatch problem after running mc cp command this issue is filed for and the version of the mc you see it with is what? Released mc or the latest mc? You wrote in your original comment:

Version: DEVELOPMENT.GOGET
Release-tag: DEVELOPMENT.GOGET
Commit-id: DEVELOPMENT.GOGE

which tells us you've used the dev code, but now you say The release mc version does not (have) the net/HTTP error., which implies us that you've used released mc version when you saw the size issue.

Please clarify which one it is. If it was the release version, which mc release was it?

By any chance, if the original version info is incorrect/wrong, please modify and correct it.

It looks like you see a separate problem, net/HTTP error, with the latest mc code, right? Please file a separate issue for it and just make a reference here in this issue.
We assume you've used the same minio server, "VERSION: 2017-09-29T19:16:56Z" in both cases/scenarios. Please confirm if this is correct.

Thanks for all your help...

kybom · 2017-11-07T03:27:53Z

Back to the size error.I download mc from https://dl.minio.io/client/mc/release/linux-amd64/mc.
the version has the size error.
My minio version "VERSION: 2017-09-29T19:16:56Z".

harshavardhana · 2017-12-07T20:55:02Z

We are not able to reproduce this on our end @kybom

ebozduman · 2018-01-31T00:45:55Z

@kybom , unless you have more information, we close this issue for the time being, since we couldn't reproduce it and make any progress on it.
Please reopen this issue or open up a new one if you can reproduce it with a different file.

harshavardhana · 2019-02-04T23:57:29Z

Re-opening as I was able to reproduce a behavior where mc cp would progress further than the Total amount , this happens upon a retry request progress bar keeps moving forward.

/etc/hosts:                  896 B / 224 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 400.00% 549 B/s -1s^C

harshavardhana · 2019-02-04T23:58:04Z

Sent the PR minio/minio-go#1070 we need also mc change for this.

harshavardhana · 2020-01-20T09:45:07Z

This is fixed already minio/minio-go#1070

ebozduman added priority: medium triage labels Oct 26, 2017

ebozduman added this to the Current milestone Oct 26, 2017

ebozduman self-assigned this Oct 26, 2017

ebozduman changed the title ~~MC cp size is bigger than real zise~~ MC cp size is bigger than real size Oct 31, 2017

kybom closed this as completed Nov 4, 2017

kybom reopened this Nov 4, 2017

harshavardhana removed the Hacktoberfest label Dec 7, 2017

ebozduman closed this as completed Jan 31, 2018

harshavardhana removed the triage label Jan 31, 2018

harshavardhana reopened this Feb 4, 2019

harshavardhana assigned harshavardhana and unassigned ebozduman Feb 4, 2019

harshavardhana closed this as completed Jan 20, 2020

harshavardhana added the fixed label Jan 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MC cp size is bigger than real size #2297

MC cp size is bigger than real size #2297

kybom commented Oct 26, 2017

vadmeste commented Oct 26, 2017 •

edited

ebozduman commented Oct 31, 2017 •

edited

ebozduman commented Nov 2, 2017 •

edited

kybom commented Nov 4, 2017

kybom commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

kybom commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

kybom commented Nov 6, 2017

harshavardhana commented Nov 6, 2017

ebozduman commented Nov 6, 2017 •

edited

kybom commented Nov 7, 2017 •

edited

harshavardhana commented Dec 7, 2017

ebozduman commented Jan 31, 2018 •

edited

harshavardhana commented Feb 4, 2019

harshavardhana commented Feb 4, 2019

harshavardhana commented Jan 20, 2020

MC cp size is bigger than real size #2297

MC cp size is bigger than real size #2297

Comments

kybom commented Oct 26, 2017

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

mc version

System information

vadmeste commented Oct 26, 2017 • edited

ebozduman commented Oct 31, 2017 • edited

ebozduman commented Nov 2, 2017 • edited

kybom commented Nov 4, 2017

kybom commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

kybom commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

harshavardhana commented Nov 4, 2017

kybom commented Nov 6, 2017

harshavardhana commented Nov 6, 2017

ebozduman commented Nov 6, 2017 • edited

kybom commented Nov 7, 2017 • edited

harshavardhana commented Dec 7, 2017

ebozduman commented Jan 31, 2018 • edited

harshavardhana commented Feb 4, 2019

harshavardhana commented Feb 4, 2019

harshavardhana commented Jan 20, 2020

vadmeste commented Oct 26, 2017 •

edited

ebozduman commented Oct 31, 2017 •

edited

ebozduman commented Nov 2, 2017 •

edited

ebozduman commented Nov 6, 2017 •

edited

kybom commented Nov 7, 2017 •

edited

ebozduman commented Jan 31, 2018 •

edited