Asynchronous assertions support

[fatuhoku]

Sometimes it's useful to test that a property eventually becomes true within a set timeout period, especially for slightly longer-running integration tests.

This is particularly useful for making assertions when using Pykka, a concurrency library based on the actor model.

In Objective-C, the default BDD test framework of choice Kiwi already comes with a couple of useful async assertion primitives out of the box; sure should too!

[fatuhoku]

Well, I've written such an assertion function.

def assert_eventually(assertion, timeout=5, interval=0.05):
    assert interval < timeout
    start = datetime.now()
    diff = timedelta(seconds=timeout)
    while True:
        try:
            assertion()
            return
        except:
            if datetime.now() - start > diff:
                raise AssertionTimeoutError("assertion didn't succeed after {0} seconds".format(timeout))
            time.sleep(interval)
            logging.debug("assert_eventually wait finished after %s seconds", datetime.now() - start)

Of course the error handling logic could be a tad more helpful if it reported what the last assertion failure really was within assertion() rather than just saying not all assertions were valid after a particular period of time.

Usage:

x = False
# Start a thread that executes asynchronously that sets the name x to True.
assert_eventually(lambda: x.should.be.true)

It'd be really great if this could work its way into sure such that we can write something more like

x = False
# Start a thread that executes asynchronously that sets the name x to True.
x.should.eventually.be.true  # !!! Note the awesome readability

Perhaps you would also have some syntax ideas for making multiple assertions as clean as possible?

[gabrielfalcao]

I like that idea.

Maybe we could have your assertion with an interface like this:

@expect.within(5).seconds
def x_should_be_true():
    # check for conditions

or

with expect.within(5).seconds:
    # check for conditions

What you think?

[fatuhoku]

Hey @gabrielfalcao. Great! We should have both multiline and inline forms:

Multiline form

I like the latter more than the former. I don't like decorators much.

I can totally see myself writing:

with expect.within(1).seconds:
    x.should.be.greater_than_or_equal(100)

with expect.within(5).millis:
    application.async_initialise()
    application.state.should.be.ok
    for system in application.all_systems:
        system.should.be.equal('groovy')

Or even:

with in(5).seconds:
    application.running.should.be.true

I personally LOVE this one. Despite the gaping space in between with and in, it reads well because it can either be read as

• within 5 seconds, application running should be true; quite natural
• with keyword is noise. Okay, in 5 seconds, application running.should be true... not as succinct but the meaning is kind of close.

Of course, there should be a full range of time intervals to choose from, e.g. seconds, millis, minutes, hours, days, weeks, years etc. along with their singular forms.

Inline form

In addition to the suggestions aforementioned, I've thought of some more inline forms worth considering. Which ones do you prefer?

with default, nominal timeout of 3 seconds or something:

foo.should.eventually.be.equal(bar)               # *** I prefer this one
foo.should.be.equal(bar).eventually

with custom timeout:

foo.should.within(5).seconds.be.equal(bar)      # 5/10   a bit clunky, still makes sense
foo.should.in(5).seconds.be.equal(bar)          # 4/10   again, clunky
foo.should.be.equal(bar).within(5).seconds      # 8/10   as clear as you can have it
foo.should.be.equal(bar).in(5).seconds          # 5/10   not as clear as 'within'

[gabrielfalcao]

@fatuhoku your code inspires poetry, I'm loving it!

I invite @clarete, python magician behind forbidden fruit to join the discussion. We both are constantly tinkering in sweetening the python syntax for testing purposes.

[fatuhoku]

I've recently come across a use-case for potentially long-running assertions where it's really useful to not only check for the assertion, but also provide an invariant.

I'm using VCRpy to assert on the number of HTTP interactions with a server; say, uploading a file. This takes 10 seconds to complete, so it's certainly long running.

I want to check that there is exactly one interaction by using cass.play_count.should.be.equal(1).

The problem is if I just write

assert_eventually(lambda: cass.play_count.should.be.equal(1), timeout=10.0)

... and there were two interactions, I'd have to wait for 10 whole seconds before I know about it even if it may have finished in 0.01 seconds (thanks to a canned cassette response vs. actually hitting the network).

In most cases, to keep my integration test going super fast I write:

assert_eventually(lambda: cass.play_count.should.be.equal(1),
                           invariant=lambda: cass.play_count.should.be.lower_than(2),
                           timeout=10.0)

... so that the test can fail-fast. In this case, it fails very very very fast.

The implementation is enhanced thus:

def assert_eventually(assertion, invariant=lambda:None, timeout=5, interval=0.05):
    assert interval < timeout
    start = datetime.now()
    diff = timedelta(seconds=timeout)
    while True:
        invariant()           # check the invariant before retrying the assertion 
        try:
            assertion()
        ... # the rest as before

[fatuhoku]

As for sure-esque syntax of this little extension, I would say there should just be a multiline form like so:

with in(5).seconds:
    with invariant(lambda: foo.should.be.ok):
        foo.should.be.equal(bar)

It's unclear to me how you'd stop people writing the with invariant(): block without first being enclosed by with in():.

An inline form is a bit too noisy, especially if using a lambda: looks inevitable.

[timofurrer]

I really like does ideas, too.
Was one of you @fatuhoku @gabrielfalcao ever working on it? In that sense of a branch or something were development could continue and eventually make it's way into upstream sure?
I'd like to see that! 🍻

[gabrielfalcao]

@timofurrer I have worked on something similar in the past: https://github.com/gabrielfalcao/sure/blob/master/OLD_API.md#timed-tests
It's one of sure's features that I haven't used in a long time and forgot about until now :)

As far as syntax, I have some points:

# this is beautiful but `in` is a keyword :(
with in(5).seconds:
    with invariant(lambda: foo.should.be.ok):
        foo.should.be.equal(bar)

I like this one a lot:

assert_eventually(lambda: cass.play_count.should.be.equal(1),
                           invariant=lambda: cass.play_count.should.be.lower_than(2),
                           timeout=10.0)

But to make it more sure-esque I'd make it an AssertionBuilder method that is aware of callables:

# calling the assertions below will cause the test to block until the condition is satisfied
cass.play_count.should.eventually.equal(1)
# and its counterpart
cass.play_count.should.never.equal(1)

# with timeouts
from sure import within

@within(ten=seconds):
def test():
    cass.play_count.should.within_timeout.equal(1)
    # and its counterpart
    cass.play_count.should.never.equal(1)

# that will block for 10 seconds and if the conditions are not met, then the `@within` decorator raises a timeout exception

If this feature does get implemented we need to make sure that the assertion error is very explicit and human friendly, so that debugging the error is as easy as possible.

---

Now, with all that said, this is not at all asynchronous testing. It would be nerly impossible for sure to provide a simple, out-of-the-box solution for all the async programming possibilities in python: threading, multiprocessing, gevent, tornado, put your favorite IO loop engine here

So if we do implement this feature, let's call it timed testing or some word to the effect, but this feature will not address scope problems due to different asynchronous contexts (i.e: IO engines)

As a matter of fact, sure should NEVER worry about I/O.
I have an idea to expose a plugin interface so that anyone can extend sure's assertions in their own project, and at that point a community can create a bunch of extensions for all sorts of things like:

• automatic integration with the mock library
• testing low level I/O operations
• testing sockets
• testing http requests

Finally that brings me to a final question to @fatuhoku how exactly you intend to use this feature, could you share some code that would be using this new feature?

[timofurrer]

Sounds pretty cool. I haven't used this feature either - I didn't know about it.
Yeah, the in(...)-syntax would be awesome - unfortunate.

However, the `should.eventually.equal* is pretty cool, too.

I'm curious about the use-cases...