Error when uploading to GitHub release: "Error from GitHub: Validation Failed"

[daviwil]

Hey folks, I'm running prebuild on CI after not having used it in a while and I'm getting this error when prebuild tries to upload assets to a GitHub release:

prebuild ERR! build Error: Error from GitHub: Validation Failed
prebuild ERR! build     at responseHandler (C:\projects\node-pty-prebuilt\node_modules\ghutils\ghutils.js:25:23)
prebuild ERR! build     at BufferList._callback (C:\projects\node-pty-prebuilt\node_modules\jsonist\jsonist.js:61:7)
prebuild ERR! build     at BufferList.end (C:\projects\node-pty-prebuilt\node_modules\bl\bl.js:105:10)
prebuild ERR! build     at DuplexWrapper.onend (C:\projects\node-pty-prebuilt\node_modules\hyperquest\node_modules\readable-stream\lib\_stream_readable.js:537:10)
prebuild ERR! build     at Object.onceWrapper (events.js:273:13)
prebuild ERR! build     at DuplexWrapper.emit (events.js:187:15)
prebuild ERR! build     at C:\projects\node-pty-prebuilt\node_modules\hyperquest\node_modules\readable-stream\lib\_stream_readable.js:934:16
prebuild ERR! build     at process._tickCallback (internal/process/next_tick.js:61:11)
Command exited with code 2

I've recreated the GitHub token I'm using in CI to upload the releases so it shouldn't be an authentication issue. Any idea what might be causing this?

[daviwil]

Also, I'm using Node 10 on the CI machine so that I can make prebuilds for that version, dunno if that is contributing to the issue. Strangely the issue went away on its own on TravisCI after a rebuild but it fails consistently on Appveyor.

[ralphtheninja]

@vweevers Any thoughts?

[ralphtheninja]

@daviwil We could start by tweaking ghutils to give a better error message. We might get a more meaningful error message. I'll dig a bit.

[vweevers]

We could start by tweaking ghutils to give a better error message. We might get a more meaningful error message. I'll dig a bit.

@ralphtheninja isn't "Validation failed" an error returned by GitHub? Not much we can do.

[ralphtheninja]

It is, but there might also be a .errors array with extra information that we can tack on. If present. https://developer.github.com/v3/#client-errors

[ralphtheninja]

rvagg/ghutils#16

[ralphtheninja]

@daviwil You could try AppVeyor again to see if we get any better messages.

[daviwil]

Thanks @ralphtheninja! Definitely clearer now:

Error from GitHub: Validation Failed ([{"resource":"ReleaseAsset","code":"already_exists","field":"name"}])

Looks like there are some partially uploaded assets from a previous build:

Just cleared those out and running another build to see if it resolves the issue.

[daviwil]

Failed again for the same reason... I'll try debugging it locally on my machine to see what's going on.

[reconbot]

I'm getting this for node serialport when I switched to node 10 builds to work around the prebuild/prebuildify#10 build_v8_with_gn bug

[ralphtheninja]

@daviwil Did you manage to get any more information by debugging locally?

[reconbot]

it appears to be uploading the files twice

[ralphtheninja]

Might be a bug in https://github.com/ralphtheninja/ghreleases , but strange that we haven't noticed it before. I'll try to reproduce with leveldown.

[ralphtheninja]

@daviwil I went through your setup for node-pty-prebuilt and it's quite different from how we do it on e.g. leveldown. We use prebuild-ci (by @juliangruber) and multiple node versions in e.g. .travis.yml/appveyor.yml, i.e. we always build using the corresponding node version, which means we never use node 10 to build binaries for node 9 etc.

I can explain in greater detail if you like.

[ralphtheninja]

I'm wondering if there are multiple builds running on e.g. AppVeyor that try to build and upload the same binaries.

[daviwil]

Thanks for looking into this @ralphtheninja! I hadn't come across prebuild-ci before, looks like that might solve this problem. I'll give it a shot!

[reconbot]

I can confirm that I get dupes uploaded in a single call to my upload npm script and that no other builds produce the same files. "prebuild-upload": "prebuild --all --strip --verbose"

[ralphtheninja]

I can confirm that I get dupes uploaded in a single call to my upload npm script and that no other builds produce the same files. "prebuild-upload": "prebuild --all --strip --verbose"

So all it takes to reproduce is a npm run prebuild-upload in node-serialport? Which version of node? I'd like to reproduce this.

[reconbot]

Node 10

…

On Sat, Jun 30, 2018, 5:12 AM Lars-Magnus Skog ***@***.***> wrote: I can confirm that I get dupes uploaded in a single call to my upload npm script and that no other builds produce the same files. "prebuild-upload": "prebuild --all --strip --verbose" So all it takes to reproduce is a npm run prebuild-upload in node-serialport? Which version of node? I'd like to reproduce this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#221 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABlbnwrIK_P0ln_2yK5ggh_Q12FXS4rks5uB0EMgaJpZM4Uz3ZL> .

[reconbot]

It looks like I'm getting this on node 8.

https://travis-ci.org/node-serialport/node-serialport/jobs/406510232

prebuild ERR! build Error: Error from GitHub: Validation Failed ([{"resource":"ReleaseAsset","code":"already_exists","field":"name"}])
prebuild ERR! build     at createError (/home/travis/build/node-serialport/node-serialport/node_modules/ghutils/ghutils.js:35:10)
prebuild ERR! build     at responseHandler (/home/travis/build/node-serialport/node-serialport/node_modules/ghutils/ghutils.js:25:23)
prebuild ERR! build     at BufferList._callback (/home/travis/build/node-serialport/node-serialport/node_modules/jsonist/jsonist.js:61:7)
prebuild ERR! build     at BufferList.end (/home/travis/build/node-serialport/node-serialport/node_modules/bl/bl.js:105:10)
prebuild ERR! build     at DuplexWrapper.onend (/home/travis/build/node-serialport/node-serialport/node_modules/hyperquest/node_modules/readable-stream/lib/_stream_readable.js:537:10)
prebuild ERR! build     at Object.onceWrapper (events.js:313:30)
prebuild ERR! build     at emitNone (events.js:111:20)
prebuild ERR! build     at DuplexWrapper.emit (events.js:208:7)
prebuild ERR! build     at /home/travis/build/node-serialport/node-serialport/node_modules/hyperquest/node_modules/readable-stream/lib/_stream_readable.js:934:16
prebuild ERR! build     at _combinedTickCallback (internal/process/next_tick.js:131:7)

It does publish the binaries, but seems to fail after when maybe it tries again?

[robinwassen]

This problem seems to be in prebuild rather than any dependency.

Added some debug information and upload.js:filtered contains duplicate entries that causes some files to be uploaded twice.

posting https://api.github.com/repos/robinwassen/drivelist/releases
success /repos/robinwassen/drivelist/releases
success /repos/robinwassen/drivelist/releases/tags/v6.4.0
Uploading assets, filtered: [ 'prebuilds\\drivelist-v6.4.0-node-v46-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v47-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v48-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v51-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v57-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v59-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-node-v64-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v47-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v48-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v49-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v50-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v51-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v53-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v54-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v57-win32-x64.tar.gz',
  'prebuilds\\drivelist-v6.4.0-electron-v57-win32-x64.tar.gz' ]
success /repos/robinwassen/drivelist/releases/tags/v6.4.0
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v46-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v47-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v57-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v51-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v59-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v64-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v47-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-node-v48-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v48-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v49-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v51-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v50-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v53-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v54-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v57-win32-x64.tar.gz
posting https://uploads.github.com/repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v57-win32-x64.tar.gz
error /repos/robinwassen/drivelist/releases/12283559/assets?name=drivelist-v6.4.0-electron-v57-win32-x64.tar.gz { message: 'Validation Failed',
  request_id: '2B6D:4AC4:2653FF:2882D8:5B694B21',
  documentation_url: 'https://developer.github.com/v3',
  errors:
   [ { resource: 'ReleaseAsset',
       code: 'already_exists',
       field: 'name' } ] }

[robinwassen]

I have found the root cause of the issue.

require('node-abi').supportedTargets returns the following targets:

targets [ { runtime: 'node', target: '4.0.0', abi: '46', lts: false },
  { runtime: 'node', target: '5.0.0', abi: '47', lts: false },
  { runtime: 'node', target: '6.0.0', abi: '48', lts: false },
  { runtime: 'node', target: '7.0.0', abi: '51', lts: false },
  { runtime: 'node', target: '8.0.0', abi: '57', lts: false },
  { runtime: 'node', target: '9.0.0', abi: '59', lts: true },
  { runtime: 'node', target: '10.0.0', abi: '64', lts: false },
  { runtime: 'electron', target: '0.36.0', abi: '47', lts: false },
  { runtime: 'electron', target: '1.1.0', abi: '48', lts: false },
  { runtime: 'electron', target: '1.3.0', abi: '49', lts: false },
  { runtime: 'electron', target: '1.4.0', abi: '50', lts: false },
  { runtime: 'electron', target: '1.5.0', abi: '51', lts: false },
  { runtime: 'electron', target: '1.6.0', abi: '53', lts: false },
  { runtime: 'electron', target: '1.7.0', abi: '54', lts: false },
  { runtime: 'electron', target: '1.8.0', abi: '57', lts: false },
  { runtime: 'electron', target: '2.0.0', abi: '57', lts: false } ]

Only the runtime and the abi fields are used to build the filename and prebuild package name, which results in one file for drivelist-v6.4.0-electron-v57-win32-x64.tar.gz, while the upload function tries to upload once per target which means that electron-v57 will always be uploaded twice.

[ralphtheninja]

Only the runtime and the abi fields are used to build the filename and prebuild package name, which results in one file for drivelist-v6.4.0-electron-v57-win32-x64.tar.gz, while the upload function tries to upload once per target which means that electron-v57 will always be uploaded twice.

Awesome find! I'll start by patching ghreleases as well.

[ralphtheninja]

ralphtheninja/ghreleases@81470ee

[robinwassen]

@ralphtheninja I have submitted a PR to fix this issue on the prebuild side also: #226

Thanks! It's up to you if you want to keep a GIGO policy or implement a filter for this, because I think it is quite an invalid input to ghreleases to call it with duplicate entries.