Disable teardown when failfast is enabled

[kopecmartin]

If a test failed, sometimes I would like to look at the resources the test created and investigate why that happened. For example if a test, which creates VM and then tries to ssh and test something, fails, I'd like to try to ssh to that particular VM myself and check for example if ssh is enabled, and so on..

Is this behavior somehow supported at the moment? If not, is something like that planned in the near future?

I think it would be useful with failfast option. If this feature and failfast are enabled and a test fails, test run is stopped, teardown omitted and we can check what happened with the same resources the test had.

[freeekanayaka]

Hi,

I think this should be treated as an implementation detail of your tearDown method. For example, your TestCase object could have a flag to signal that some particular resource (the VM in your case) should not be destroyed.

There are many ways to pass flags and parameters to TestCase objects, ranging from a custom test runner and/or custom TestSuite constructor, to simple environment variables.

Unfortunately making --failfast unconditionally skip tearDown wouldn't meet all use cases (for example there could be some resources that you do want to destroy and some other that you don't).

One feature that I'd be in favor of, would be an easy way to pass parameters to test cases (e.g. the flag you need), however that's probably not easy to implement within the constraints of the Python unittest stdlib API. Perhaps @rbtcollins might have more ideas on this.

[kopecmartin]

Hi,

thank you for your quick response. Yes, my TestCase could have such a flag, but the problem is, I'd like to omit cleaning resources only if the test failed. I was tracking the current code and if I understand it correctly, there is no way, how I could find it out.
When the teardown method is called, there is a result object which holds information about a number of successful and failed tests. So I was thinking, would it be possible to access somehow the result object from a TestCase?.

[freeekanayaka]

Ah yeah, you're right. I think this is a pain point in the current unittest design. Again, @rbtcollins might hopefully shed more light on this topic.

One hack you could do is using the _get_result function from testresources (from testresources import _get_result), which essentially traverses up the stack looking for the test result object.

At that point in your TearDown you can probably do something like:

if not result.wasSuccessful():
    return # i.e. don't do any cleanup

The testresources package would be a more appropriate choice for managing this kind of testresources (like a VM), and in theory it should handle your use case via TestResource.finishedWith(), which gets passed the test result object.

However, bear in mind that while testresources is very powerful, it is also a bit complex and has some rough edges. For example:

https://bugs.launchpad.net/testresources/+bug/1506476

might affect you (although you might not need nested resources).

Hope this helps.

[kopecmartin]

For now the hack using _get_result function should be enough. Thank you, it has really helped!