Reduce CI verbosity and complexity

[saulshanabrook]

The goal with this PR is to make CI faster, easier to debug, and less flaky.

TODO

• Figure out how to get rendermine test to pass in CI when running with coverage. (removed cache loader)

Results

• Time per build reduced from ~2 hours 20 min to ~1 hour 40 minutes. This should decrease our load on Travis which will make our PRs build quicker.
• Lines printed in the JS tests decreased from ~6500 to ~600. This makes it easier to load the test output page and scroll through it, without it slowing down your browser.

Changes

• Switched all tests to run with chrome headless and use puppeteer instead of firefox on selenium. This means we don't need to download firefox or any webdrivers in CI.
• Upgrade karma
• Changed test scripts to use set -ex; set -o pipefail, instead of set -x.**
• The -e will make sure the test exits with a failure code if any of the commands do.
• The -o pipefile will make the test exit with a failure code if any piped command does.
• Added -q to pip install
• Disabled python logging for test server
• In CI, only show lerna warnings, not info messages about starting commands
• In CI hide all stdout of JS tests. This means, that test runs won't output anything, unless they fail. If they fail, lerna will output the failure to stderr, so it will still be seen in CI. See this build for an example of the failing output.
• Revert karma timeouts to default. I am finding that longer timeouts/more retries just make it harder to debug.

Future Improvements

There is still many ways to improve these metrics, but I think this is a good time to stop where we are at. Future work could include:

• Consolidating testing logic, including moving services tests with rest of tests
• Running all tests in one webpack build. This would hopefully speed up time spent type checking and building during all tests.
• Completely change how packages are organized so that it's a little less od-hoc how tests are built and dependencies are used. Maybe use webpack dll package to do this and investigate previous work done for extensions (Third Party Extensions (single build) #728).
• Move all tests to CircleCI and do tests in stages building docker images, so that code only has to be built/type checking once, instead of again and again for each test type.

[saulshanabrook]

We could also auto cancel branch builds, so that if you push new changes to a branch the old ones builds get cancelled. We often are running up against travis's max concurrent builds, so this might help with that.

[ian-r-rose]

👍 on auto-canceling builds. I thought we were already doing that, in fact.

[saulshanabrook]

I flipped the switch.

[saulshanabrook]

I actually reverted to using the non container builds and re-enabled deleting yarn. Container builds might run a bit slower anyway so I think that's fine. Issue description is up to date, but my comment you quoted isn't.

…

On Thu, Jun 28, 2018, 5:37 PM Jason Grout ***@***.***> wrote: Ah, it's too bad we can't remove the rm command for yarn. I wonder if there is some creative way to make that command unavailable without requiring sudo. The non sudo builds start up quicker. I suppose we could put something in the path to mask the system yarn that would throw an error. The point is that we want to make sure that the system yarn doesn't work, so we are testing that we are actually using our yarn, and work on a system without a working yarn. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4735 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABIZTIiIUdka9pA5H3RpwcgTLFGVkXxCks5uBUyVgaJpZM4Upx_u> .

[saulshanabrook]

The cache-loader introduced in #4698 is causing me some problems when running with the chrome browser. I am getting failure with the rendermine tests when testing in the chrome browser and using the cache loader, some of the time. When I inspect the generated JS in the browser, it obviously has some problems. Here is an excerpt of index.ts:

/**
 * A namespace for private data.
 */
var Private;
(function (Private) {
    Private.manager = new services_1.ServiceManager();
    Private.textFactory = new docregistry_1.TextModelFactory();
    Private.notebookFactory = new notebook_1.NotebookModelFactory({});
    var JSONRenderer = /** @class */ (function (_super) {
        __extends(JSONRenderer, _super);
        function JSONRenderer() {
            var _this = _super !== null && _super.apply(this, arguments) || this;
            _this.mimeType = 'text/html';
            return _this;
        }
        JSONRenderer.prototype.renderModel = function (model) {
            var source = model.data['application/json'];
            model.setData({ data: { 'text/html': json2html(source) } });
            return _super.prototype.renderModel.call(this, model);
        };
        return JSONRenderer;
    }(rendermime_1.RenderedHTML));
    var jsonRendererFactory = {
        mimeTypes: ['application/json'],
        safe: true,
        createRenderer: function (options) {
            debugger;
            return new JSONRenderer(options);
        }
    };
    Private.rendermime = new rendermime_1.RenderMimeRegistry({
        initialFactories: rendermime_1.standardRendererFactories
    });
    Private.rendermime.addFactory(jsonRendererFactory, 10);
})(Private || (Private = {}));

This file is messed up because JSONRenderer is transpiled to a function, but it still called with new JSONRenderer, like it is a class. If we take away the cache-loader, it looks like this:

/**
 * A namespace for private data.
 */
var Private;
(function (Private) {
    Private.manager = new services_1.ServiceManager();
    Private.textFactory = new docregistry_1.TextModelFactory();
    Private.notebookFactory = new notebook_1.NotebookModelFactory({});
    class JSONRenderer extends rendermime_1.RenderedHTML {
        constructor() {
            super(...arguments);
            this.mimeType = 'text/html';
        }
        renderModel(model) {
            let source = model.data['application/json'];
            model.setData({ data: { 'text/html': json2html(source) } });
            return super.renderModel(model);
        }
    }
    const jsonRendererFactory = {
        mimeTypes: ['application/json'],
        safe: true,
        createRenderer(options) {
            debugger;
            return new JSONRenderer(options);
        }
    };
    Private.rendermime = new rendermime_1.RenderMimeRegistry({
        initialFactories: rendermime_1.standardRendererFactories
    });
    Private.rendermime.addFactory(jsonRendererFactory, 10);
})(Private || (Private = {}));

Which is more reasonable.

The errors might be with with combing it with the thread-loader (webpack-contrib/cache-loader#20).

I am going to remove it here and see if the tests still run in an acceptable amount of time.

[saulshanabrook]

@jasongrout This is ready for review again.

[blink1073]

Looks great, just one comment

[blink1073]

Intentional?

[saulshanabrook]

Nope, not intentional. Good catch. Thank you for taking a look over this.

[afshin]

Thanks for working on this @saulshanabrook! This sort of plumbing is both crucial and frustrating.