All notable changes to this project will be documented in this file.
- Logging Support: Put Test name into Logging MDC to make it easy to see what was logged by what test. See included logback.xml file for an example how to use it.
- Send executionStarted event for failed contexts. Idea will ignore executionFinished events for not started tests so the error message would not appear in the test runner. (it would appear in the console output of the test runner)
- Fix some edge cases with the new junit engine.
- Fix running "all tests in project" by adding a fake test, because idea does not tests when the initial test plan is empty
- Improve handling of duplicate root context names by making them unique by adding a number(tests, tests-1, tests-2)
- Default to the new junit engine. you can still use
-Dfailgood.new.junit=falseto use the old one
Highlights of 0.9 are the new junit engine and the extended support for fixtures via given
There are also new deprecations while the api is fine-tuned in response to using failgood in all kinds
of test suites.
Use idea inspect code to automatically replace deprecated versions.
- afterEach and dependency are no longer available in the TestDSL, where it makes no sense to have them.
- tests are now created via
val tests = tests { ... }orval tests = testsFor("...") {}. top leveldescribefunctions are deprecated. Use idea inspect code to automatically replace deprecated versions. - Given is now accessed via a
givenproperty instead of a parameter to the test function - Given block can now lazily access the parent context
- Private test classes now throw an error instead of being ignored. Now that we find tests by Annotation, it makes no sense to mark a class as a Test and make it private.
- The Root context name now contains the name of the test file when it makes sense
- Added SuspendAutoCloseable interface with a suspend close function and support it in the autoClose method
- The Root context can now also have a given block
- New junit platform engine that is totally async To try it out add -Dfailgood.new.junit=true to your test run or set it in junit-platform.properties. It works pretty well and will become the default before 1.0
- Remove all deprecated methods
Ok another bugfix release before 0.9
- Don't report ignored tests that should not run (#160)
- Ignore tests that can not be instantiated in pitest plugin
This version will be the last before 0.9 where all deprecated methods will be removed. 0.9.x versions will be beta/release candidate versions for 1.0
- Sub-contexts can now also be ignored
- New Ignore API (#95,#63,139)
- AssertHelpers.containsExactlyInAnyOrder works when elements are not Comparable
- Handle exceptions in dependency block as result. (#130)
- Don't crash when root context is flaky.
- Correctly handle root context timeouts. the timeout should be for one test execution that is part of the Root Context and not for all the tests that are executed as part of the root context.
- allow to disable isolation for sub-contexts.
- fix running tests by uniqueid where the root context contains ( or ) (#109)
- handle exceptions in afterEach after a test failure (fixes #110)
- context silently ignored when using describe for subcontexts. (#104)
- Running tests using FailGood::runAllTests or using the auto-test feature fails on Windows (#114)
- Run single tests from gradle (#96)
- Collect Events that happen during test execution and add them to an exception in case of error. Or print it to a file when debug is set.
- Failgood incorrectly reports error when only a single test is selected that is not a failgood test (#93)
- Lots of changes to the previously undocumented mock library.
- Don't rely on class name patterns and instead run every class with a failgood.Test annotation
- Support afterEach callbacks that run after each test.
- Integrate kover plugin, publish code coverage to codecov
- fix that some test names would not be found via uniqueid
- Correctly handle tags on tests created with
it
- Print a useful error message when test are not found because the test names change. (#61)
- Add support for given. Contexts can now define a function whose result is passed to tests
- Experimental run tests by tag. describe,it,test and context now have a
tagsparameter, and you can run a subset of tests by setting the env variable FAILGOOD_TAG. Currently, only contexts that are in the root can be tagged, tags in subcontexts of subcontexts are currently not found - When a test fails print its uniqueId for easy re-running of that test from idea (just create a junit run config that runs by uniqueId)
- Issue a warning when tests are created in the wrong context because a method has the wrong receiver
- Catch errors in the junit engine, and log them to stdout. Tell users to submit an issue
- Print slowest test after test run when env var PRINT_SLOWEST_TESTS is set in the junit engine
- Treat test timeouts as failed test instead of throwing and failing the whole suite (#73)
- ignore errors in failed contexts (fixes #38)
- Errors in close callbacks were not caught correctly after a test failure (#65)
- Correctly handle errors in contexts and root context, for example duplicate names (#58)
- Report exceptions in autoClose callbacks as test failure (#53)
- don't throw an error when the junit engine finds no tests, to allow other engines to run their tests. (#48)
- add support for package filtering. this fixes running all tests in a package from IDEA. (#50)
- support contexts defined in other variables than
context. This also enables multiple contexts in one class. (#49)
- Added support for root contexts without isolation. (experimental, API will probably change)
- Call autoClose tear down methods for failed tests too. Fascinating that this huge bug was undetected so long. #43
- Use the same timeout for context evaluation and test execution (configurable via env var TIMEOUT in milliseconds) (#32)
- Report timeouts in context evaluation as Failed Context instead of throwing and failing the whole suite. (#36)
- Correctly report errors in root contexts even in gradle. (#34)
- bump timeout to 40 seconds for now.
- Output UniqueIds of failed tests in junit engine runner to make it possible to re-run them in IDEA.
- Fix some run-tests-by-uniqueId edge cases
- Support UniqueId junit selectors to run single tests or subcontexts. This already works in idea if you select uniqueId
in the run config and enter the uniqueId selector like
this:
[engine:failgood]/[class:The+ContextExecutor(failgood.internal.ContextExecutorTest)]/[method:with+a+valid+root+context]/[method:reports+line+numbers]( with + instead of space). Now idea just needs to do this automatically:https://youtrack.jetbrains.com/issue/IDEA-277855
- Add a runtimeOnly dependency on junit launcher to make it easier to run test in idea.
- Correctly handle the case where duplicate root contexts contain tests with the same name.
- Improve running the tests in Gradle. It works pretty well now.
- Report failures in root contexts instead of throwing and failing the whole suite
- Introduce @Test annotation to replace junit platforms @Testable. This will probably change again before 0.5.0
- Mocks: Replace
thenReturn(result)withthen {result}, and allow mock methods to throw exceptions
- Use kotlin 1.5.21. Coroutines 1.5.1
- Support mocking functions with a nullable return type.
- Duplicate root context names no longer break the junit engine. It's still probably a good idea to give your root contexts unique names. But the JUnit engine will now work even if you name all your root contexts RootContext. (or Rudolph)
FailGood.runTest()andFailGood.runAllTests()are nowsuspend, so you have to make your test main functionssuspendtoo.
- Allow to override the context timeout by setting the env variable TIMEOUT. Set it to any millisecond value to change it or to something that is not a number to disable timeouts.
- Allow ordering of contexts by passing an order value. contexts with lower order will be executed first.
afterSuite {}method that is called after all tests have finished.
- Fix println which was not doing anything in the JUnit Platform engine.
- Fix stack overflow when running a single test that uses println.
- Change the project name to FailGood.
- Run a single test class from idea even if it is not called *Test if it's annotated with @Testable
- Support declaring test contexts in classes. IDE support for tests in classes is just better. You can now add a @Testable annotation to your test class and run a single class from idea. The class is only instantiated once, so don't put dependencies into the class outside a context.
- Correctly report failing contexts in the JUnit platform engine.
- Support autoClose inside tests.
- Add autoClose(AutoCloseable) to close AutoCloseables without specifying a close callback.
- Mocking library
call(..)helper now supports suspend functions printlnmethod inside a test case for thread safe test logging. Currently, the output is only available in the JUnit platform engine, The main test runner does not yet display it.- Use DSLMarker to ensure that tests do not create contexts. This is never what you want, so failgood now protects you
from it. Here is an example:
test("tests can not contain nested contexts") { context("this should not even compile") {} // compile error }
- Mocking. Failfast now has its own mocking framework. It's very simple and can only mock interfaces, but it's much faster than mockk, and has a nice api. (Look in the kdoc of Mock.kt or MockTest.kt for more info)
- Allow declaring multiple root contexts in one class file. This is especially useful to create root contexts dynamically, for example to run the same test cases on multiple databases.
- Classes that fit the failgood naming pattern *Test but contain no failgood context are now ignored. This makes it possible to convert bigger test suites that still use other frameworks without converting everything at once.
- Rename itWill to pending
- AutoCloseables were closed in the same order they were created. Now they are correctly closed in reverse order.
- Report individual Test duration correctly. Before the first test in each context had a wrong time reported.
- Make autoClose callback method suspend for easier closing of resources that need coroutines.
- Output tests/sec in test summary.
- Correctly report failures in contexts, even when the first context run is successful.
- Send test start and stop events in correct order
- Always report contexts as Successful, as requested by the Junit Platform API
- JUnit Platform Engine for easy and comfortable test running from IDEA. It's new and experimental, but it works pretty well. For some suites it can take pretty long for the test tree to appear, this will be improved soon.
- Print each context result as soon as the context is finished.
- Display test failures in a more compact way. Stack traces are shortened to only show lines from the test, and the exception name is not added to the output when it is an assertion error. Those usually have an error message that speaks for itself
- Color output in test report (robfletcher)
- A changelog
- Display a stacktrace entry that leads to the test method for failed tests.
- Report errors in contexts as failing tests.
- Tests are reported in the same order as they appear in the file.
- Contexts are reported in the same order as they appear in the file. who would have thought that they were reported in random order before.
- Publish to maven central. goodbye jcenter, you will be missed
- Don't output slow tests and pending tests separately for now. need to find a more useful way of showing this
- Duplicate contexts fail the suite. failgood always needed contexts and tests to have unique names. before duplicate tests would just be ignored, now this is detected
- Contexts with the same name as a test in the same context fail the suite.
- Fixed a threading bug that only appears on the OpenJ9 JDK.