Properly compare test files between jobs

[jbhoosreddy]

Summary of changes

• With the help of the previous PR Prevent comparision on CircleCI master #16, I've been able to compare a master branch and my PR branch artifacts (by downloading them locally) and figuring out what are the remaining issues.
  • As you can see, it is correctly erroring on test failures my PR is causing.
  • I'm guessing the success values are coming from the fact that my babel PR is a month old and something else might have been fixed along the way. Now, I'll update my main PR and see if these errors remain.

But this is great progress.

➜ CIRCLE_BRANCH=PR/1234 node lib/compare-results master.tap test262.tap | $(npm bin)/tap-merge
TAP version 13
#  CIRCLECI = undefined
#  CIRCLE_BRANCH = PR/1234
#  is CircleCI master branch job: undefined
not ok 1 test/language/expressions/arrow-function/prototype-rules.js default # (expected success, got runtime error)
  ---
  name: Test262Error
  message: 'Expected SameValue(«true», «false») to be true'
  stack: |
    Test262Error: Expected SameValue(«true», «false») to be true
    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 2 test/language/expressions/arrow-function/prototype-rules.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: 'Expected SameValue(«true», «false») to be true'
  stack: |
    Test262Error: Expected SameValue(«true», «false») to be true
    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 3 test/language/expressions/arrow-function/throw-new.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: Expected a TypeError to be thrown but no exception was thrown at all
  stack: >
    Test262Error: Expected a TypeError to be thrown but no exception was thrown at
    all

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 4 test/language/expressions/arrow-function/throw-new.js default # (expected success, got runtime error)
  ---
  name: Test262Error
  message: Expected a TypeError to be thrown but no exception was thrown at all
  stack: >
    Test262Error: Expected a TypeError to be thrown but no exception was thrown at
    all

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
ok 5 test/language/literals/regexp/u-invalid-optional-lookbehind.js default # (parser error)

ok 6 test/language/literals/regexp/u-invalid-optional-lookbehind.js strict mode # (parser error)

ok 7 test/language/literals/regexp/u-invalid-optional-negative-lookbehind.js default # (parser error)

ok 8 test/language/literals/regexp/u-invalid-optional-negative-lookbehind.js strict mode # (parser error)

ok 9 test/language/literals/regexp/u-invalid-range-lookbehind.js default # (parser error)

ok 10 test/language/literals/regexp/u-invalid-range-lookbehind.js strict mode # (parser error)

ok 11 test/language/literals/regexp/u-invalid-range-negative-lookbehind.js default # (parser error)

ok 12 test/language/literals/regexp/u-invalid-range-negative-lookbehind.js strict mode # (parser error)

not ok 13 test/language/computed-property-names/class/method/symbol.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: >-
    `compareArray(Object.getOwnPropertyNames(C.prototype), ['constructor', 'a',
    'c'])` returns `true`
  stack: >
    Test262Error: `compareArray(Object.getOwnPropertyNames(C.prototype),
    ['constructor', 'a', 'c'])` returns `true`

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
ok 14 test/language/literals/regexp/named-groups/invalid-identity-escape-in-capture-u.js default # (parser error)

ok 15 test/language/literals/regexp/named-groups/invalid-identity-escape-in-capture-u.js strict mode # (parser error)

ok 16 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-2-u.js default # (parser error)

ok 17 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-2-u.js strict mode # (parser error)

ok 18 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-u.js default # (parser error)

ok 19 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-u.js strict mode # (parser error)


0..0

[jbhoosreddy]

Working in babel/babel#10579

[jbhoosreddy]

I'm trying to get the consistent file name from this title
not ok 1 test/language/expressions/arrow-function/prototype-rules.js default # (expected success, got runtime error) ---

[jbhoosreddy]

My last commit was driven by some weirdness I was noticing in test results:

"unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---*/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m"
 stack: "Consider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nSyntaxError: unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nagents.withAgent (/Users/jbhoosreddy/Projects/babel-test262-runner/lib/run-tests/index.js:62:11)\nSyntaxError: unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---*/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nParser.raise (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:6387:17)\nParser.assertModuleNodeAllowed (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10040:12)\nParser.parseStatementContent (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10011:16)\nParser.parseStatement (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:9900:17)\nParser.parseBlockOrModuleBlockBody (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10476:25)\nParser.parseBlockBody (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10463:10)\nParser.parseTopLevel (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:9829:10)\nParser.parse (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:11341:17)\nparse (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:11377:38)\nparser (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/core/lib/transformation/normalize-file.js:166:34)"

[nicolo-ribaudo]

LGTM, feel free to merge!