Test retry() and wait()

[paulmelnikow]

The retry and wait functionality are not covered by tests. Without coverage, we risk breaking them during refactors.

[paulmelnikow]

I don't think there are tests that cover the timeout handler, either.

[paulmelnikow]

retry() is covered by #71!

[Fishbowler]

Does this fix the mocha vs IcedFrisby mismatch with timeouts?

A trimmed version of my test:

describe('Our giant API', function(){

    before(function(){
        frisby.globalSetup({
            request: {
                headers: {
			'Content-Type': 'application/json; charset=utf-8'
		},
                followRedirect: false,
                json: true
            },
            timeout: 5000,
            retry: 2
        });
    })

    it('should respond with a well-formed set of content @monitoring', function(){
        const testname = this.test.title
        frisby.create(testname + ' - All GB Content')
            .get(myAPIurl)
            .expectStatus(200)
            .expectHeader('Content-Type','application/json; charset=utf-8')
            .expectJSONTypes({
                //Many, many lines of Joi
            })
        .toss();
    })
})

My npm script is this

"monitoring": "mocha specs/*/*_spec.js --fgrep @monitoring --timeout 5000"

This endpoint has a large (~500KB) payload, generated from a few different sources on demand and so takes ~4 seconds to generate (hence being behind CloudFront in production).

When it takes slightly longer I'm getting 1 run of 5s and the error of Error: Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. and no retries. If I set the mocha timeout to be much larger, then retries work as expected. Is this even a legit problem, or one of config?

[paulmelnikow]

Yes, I think it will, though it hasn't been shipped yet. We have some breaking changes and should get ready for a 2.0.0 release. Would shipping a prerelease version make sense?

[Fishbowler]

Let's hold off for now. I can get around it with a mocha commandline timeout of the timeout x retries, plus a little for processing. In my example above, I had a 5s timeout and 2 retries, so gave a generous 20s to mocha, and all is well. We haven't had any other reports of being blocked (and the workaround is now documented 😁).

A prerelease sounds like a good idea in general though. Get an alpha out to test with, once everything we want is into master. A little off-topic for this issue, but do we have a list of things we want to get into v2 before it ships?