Allure Framework Team would like to support Open Test API

[baev]

Hi there!

I am a creator of Allure Framework. Our team would like to support this project. It would be nice to create an open standard for testing.

Please feel free to contact our team to reach feedback.

Dmitry Baev

[marcphilipp]

We'd be happy to collaborate with you! Do you have concrete ideas what functionality should be added? 😄

Right now this library contains a few exceptions that are intended to be reusable by many testing frameworks, not just JUnit. I assume that would be interesting for Allure, e.g. to show expected and actual values for test failures.

[baev]

Do you have concrete ideas what functionality should be added?

I think you need to write a document which contains a list of test statuses with detailed descriptions. An example - what exactly your TestSkippedException means for the test? Is test was skipped because some features wasn't implemented yet, or user choose to don't run such test (this mean some features won't be tested and may be bugged) or test can't be runned (eg test database is not available). What did you think?

[baev]

@marcphilipp any updates on this?

[volkovs]

@baev do you mean you would like to have standard test execution statuses (for reporting systems) to be included in opentest4j (like PASSED, BROKEN, FAILED, SKIPPED, ABORTED) and their mapping from exceptions?

I guess JavaDoc already describes the reason of exceptions. E.g. TestAbortedException saying "a test was aborted during execution (e.g., due to a failed assumption)" meaning precondition or setUp method failed. TestSkippedException saying "test was skipped prior to execution (e.g., disabled or ignored)" meaning test was (temporarily) turned off.

Would you like:

• to extend exception descriptions
• to cover some other cases when test execution is incomplete or
• to add enumerate test execution standard results (so reporting tools developers could reuse standard statuses and rely on it creating their UI layouts)?

@marcphilipp would you like to have pull request on test execution statuses or to discuss it briefly previously?

[marcphilipp]

@volkovs Thanks, but I don't think this is ready for a pull request, yet. I'm not sure just having an enum (?) with the statuses would be useful. Who would use it and how?

[volkovs]

Hi @marcphilipp,

Standard exceptions should be enough for Test Execution / Report tools. But test execution status is relied on in tools collecting and aggregating test execution (could be different test frameworks) results like Sonar, Test Management Systems, probably in conditions controlling CD pipeline, as well as in custom listeners for Test Framework (onTestFailed(), onTestSkipped(), etc.) or names of according messages. And everywhere there are slight differences in naming and meaning.

I agree that unification of test execution result could be too early. Moreover enum data type doesn't match hierarchy of exceptions (since Java enum can't be extended). So, just an idea to discuss. There could be a need for it.

[mcahornsirup]

@baev is this implemented with allure2 (which would be great!) Because JUnit 5.9.0 supports this as well ... and it would make the working with reports simpler due to its unification.