Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local checksum not match remote while node-gyp install #693

Closed
2dxgujun opened this issue Aug 18, 2015 · 25 comments
Closed

local checksum not match remote while node-gyp install #693

2dxgujun opened this issue Aug 18, 2015 · 25 comments

Comments

@2dxgujun
Copy link

#645

@daveyarwood
Copy link

We're seeing the same issue -- see travis-ci/travis-ci#5562

@bnoordhuis
Copy link
Member

What's the issue? It downloads the tarball and the checksum fails? What makes you think it's node-gyp and not e.g. a network problem?

@daveyarwood
Copy link

Quoting @klemmster in that issue:

TLDR: It's a node-gyp issue, yes. I don't think there's anything you could do. (Other than making node_jsversion variable (in travis.yml) available even though language is not set to nodejs. That would make choosing a different node version easier.)

For some reason, node-gyp wasn't calculating the correct sha256sum of the downloaded node-v0.10.xx.tar.gz
Thanks to the ssh access I got from you, I was able to pinpoint the problem to
node-gyp configure.
Using NODE_DEBUG=request node-gyp configure it seems the file received was fine (at least the filesize was ok), so it doesn't seem to be a networking/proxy or caching issue. (I curled over the download a 100 times, file was also ok)
I wasn't able to pinpoint what's the root cause of the issue though. Something within install.js in node-gyp(v3.0.3).

@bnoordhuis
Copy link
Member

That it's the correct file size doesn't mean they're the right bytes; chunks may have gotten corrupted in transit. What does sha256sum (the command line tool) print for the tarball?

I'm inclined to believe it's an issue local to your environment. If everyone was hit by this, we'd be swamped in bug reports, but we're not.

@klemmster
Copy link

for i in {1..100}; do  curl -s "https://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz" | sha256sum; done

printed the correct sha256sum a hundred times

Also,

fs.readFile('/tmp/node-v0.10.36.tar.gz', function(err, data) {
  if(err) {throw err;} 
  var shasum = crypto.createHash('sha256');
  shasum.update(data); 
  console.log(shasum.digest('hex'));
});

did.

I wasn't able to find the tarball download by node-gyp. If you could point me to it, I'd be happy to provide a checksum for that as long as travis debugging instance is still available to me (don't know when they close it)

@bnoordhuis
Copy link
Member

v0.10.36 is the tarball you want? What happens when you rm -rf $HOME/.node-gyp and then run node-gyp install --dist-url=https://nodejs.org/dist v0.10.36? Please post the complete output.

@klemmster
Copy link

$ node /home/travis/.nvm/v0.10.36/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js install --dist-url=https://nodejs.org/dist v0.10.36

gyp info it worked if it ends with ok
gyp info using node-gyp@3.0.3
gyp info using node@0.10.36 | linux | x64
gyp http GET https://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz
gyp http 200 https://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz
gyp http GET https://nodejs.org/dist/v0.10.36/SHASUMS256.txt
gyp http 200 https://nodejs.org/dist/v0.10.36/SHASUMS256.txt
gyp WARN install got an error, rolling back install
gyp ERR! install error 
gyp ERR! stack Error: node-v0.10.36.tar.gz local checksum 0064afb1eafc4ddfc456036d3848986e9f101122bf9ab5039d9a4958554adac8 not match remote b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81
gyp ERR! stack     at deref (/home/travis/.nvm/v0.10.36/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:300:20)
gyp ERR! stack     at IncomingMessage.<anonymous> (/home/travis/.nvm/v0.10.36/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:339:13)
gyp ERR! stack     at IncomingMessage.emit (events.js:117:20)
gyp ERR! stack     at _stream_readable.js:944:16
gyp ERR! stack     at process._tickCallback (node.js:442:13)
gyp ERR! System Linux 2.6.32-042stab108.8
gyp ERR! command "node" "/home/travis/.nvm/v0.10.36/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "install" "--dist-url=https://nodejs.org/dist" "v0.10.36"
gyp ERR! cwd /home/travis
gyp ERR! node -v v0.10.36
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok 

@klemmster
Copy link

And the same with --verbose
http://pastebin.com/tNzqP6Vu

@bnoordhuis
Copy link
Member

Thanks. Can you try downloading v0.10.36 again but now with node v4.2.6 or v5.x as the binary that runs node-gyp? Please delete the .node-gyp directory first.

@bnoordhuis
Copy link
Member

Also, FWIW, it works perfectly for me with v0.10.41 and v6.0.0-pre (i.e. node master.)

@klemmster
Copy link

$ node /home/travis/.nvm/versions/node/v5.5.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js install --dist-url=https://nodejs.org/dist v0.10.36
gyp info it worked if it ends with ok
gyp info using node-gyp@3.0.3
gyp info using node@5.5.0 | linux | x64
gyp http GET https://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz
gyp http 200 https://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz
gyp WARN install got an error, rolling back install
gyp ERR! install error 
gyp ERR! stack Error: unexpected end of file
gyp ERR! stack     at Zlib._handle.onerror (zlib.js:363:17)
gyp ERR! System Linux 2.6.32-042stab108.8
gyp ERR! command "/home/travis/.nvm/versions/node/v5.5.0/bin/node" "/home/travis/.nvm/versions/node/v5.5.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "install" "--dist-url=https://nodejs.org/dist" "v0.10.36"
gyp ERR! cwd /home/travis/.nvm/versions/node/v5.5.0/lib/node_modules/npm/node_modules/node-gyp
gyp ERR! node -v v5.5.0
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok 

@bnoordhuis
Copy link
Member

The "unexpected end of file" error means something went wrong while downloading. It's probably either a corrupted chunk or the connection was closed prematurely.

@klemmster
Copy link

Running that little script:

var http = require('http');
var fs = require('fs');

var file = fs.createWriteStream("node-v0.10.36.tar.gz1");
var request = http.get("http://nodejs.org/dist/v0.10.36/node-v0.10.36.tar.gz", function(response) {
  response.pipe(file);
});

a 100 times:
for i in {1..100}; do rm node-v0.10.36.tar.gz1 && node download.js && sha256sum node-v0.10.36.tar.gz1; done
I do get:

…
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
cad7af68d21457d9408002394ecb0cedf46b8f858d0159b6776dcb8982289990  node-v0.10.36.tar.gz1
…
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
4a67cbef70ad6399ef64ccd4812d16fe1dde5c3680714408c1d90bef10b38fa2  node-v0.10.36.tar.gz1
b56c3a746a0502ed2e2d974c3466fb374b8a680961da63a10ee08a3dbc9f2de4  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
…
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
51de26b504db37ed79f37a8613c36dff7c6a39c4e909fe05173f1fd18c19c5d4  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
…
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
562c5e8eb639250523c198d006d8d86ca08bc269f854c557ceb7ba84a08f0d9e  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
…

while it still never fails with curl.
So the question remains why node receives corrupted chunks and closes the connection where curl doesn't.

@bnoordhuis
Copy link
Member

Your guess is as good as mine. I ran it in batches of 10 downloads a few times but I always get the right checksum:

$ for i in {1..10}; do rm -f node-v0.10.36.tar.gz1 && out/Release/node tmp/issue693.js && sha256sum node-v0.10.36.tar.gz1; done
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1

I did notice that node v0.10.40 uses a ton more CPU time than v6.0.0-pre but that doesn't seem to affect the download results:

$ for i in {1..10}; do rm -f node-v0.10.36.tar.gz1 && ../v0.10/out/Release/node tmp/issue693.js && sha256sum node-v0.10.36.tar.gz1; done
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1
b9d7d1d0294bce46686b13a05da6fc5b1e7743b597544aa888e8e64a9f178c81  node-v0.10.36.tar.gz1

My working hypothesis is still that it's an issue local to your environment.

@hintjens
Copy link

On a newly-configured Windows 10 box , node-gyp install dies at different points, with checksum errors. I ran @klemmster's download.js to see what it did. Surprise: almost every time, it produces different results. Here's five runs in a rwo (file size in bytes):

14,385,078 node-v0.10.36.tar.gz1
14,410,899 node-v0.10.36.tar.gz1
14,410,899 node-v0.10.36.tar.gz1
14,397,978 node-v0.10.36.tar.gz1
14,407,382 node-v0.10.36.tar.gz1

The correct size is 14,410,899 (running download.js on Linux shows this). So it looks like the last block is being arbitrarily lost. Whatever the cause, it makes node-gyp unusable on Windows.

@bnoordhuis
Copy link
Member

/cc @nodejs/platform-windows - is anyone able to reproduce?

@hintjens
Copy link

FWIW it is not a node-gyp issue, pretty sure of that. I also tried node-gyp-install and got the same results. Using 'node-gyp configure' I was after about 8 attempts able to get an OK, and then build. My test server is on Rackspace and has no proxy. However it does break node-gyp badly so is worth investigating IMO.

@rvagg
Copy link
Member

rvagg commented Feb 18, 2016

@hintjens (and others able to repro this) can you npm ls inside the node-gyp directory and see what version of request you have bundled with it? The download is almost entirely handled by request so the problem likely there or possibly in core (it's also interesting to note you're using v0.10.36 so perhaps it's an old bug that's since been fixed in Node v4+).

Cruising through the request issues shows things up like request/request#1568 so the problem could easily be there too.

@hintjens
Copy link

node-gyp is using: request@2.69.0

@rvagg
Copy link
Member

rvagg commented Feb 18, 2016

well then .. that's the latest, I'll file an issue over there and see if they have a clue, we can't rule out a v0.10 problem, however.

@rvagg
Copy link
Member

rvagg commented Feb 18, 2016

actually, I'm going to hold off and try a bit harder to reproduce it myself, there's some old style streams work going on in there that I think could be improved

@rgbkrk
Copy link

rgbkrk commented Feb 29, 2016

I never have this problem on Linux or OS X, almost always have this problem with Windows (now that I'm doing prebuilds for all platforms of native dependencies).

@chadwhitacre
Copy link

Fwiw, in eight Travis builds we are seeing five checksum mismatches, one "Completion callback never invoked!", one 522, and one success, suggesting that this is network-related.

@MylesBorins
Copy link
Member

MylesBorins commented Dec 9, 2016 via email

@rvagg
Copy link
Member

rvagg commented Jun 20, 2019

closing due to staleness

@rvagg rvagg closed this as completed Jun 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants