TCK: some checks could be extended to accept shorter Publishers

[akarnokd]

In PublisherVerification, many tests require 3, 5, 6, 10 or even 20 element Publishers and are not run against, for example 0 or 1 Publishers (for example, 309, 317, etc).

I think these tests could be extended to support more length but I don't know what's the policy about adding more tests to PublisherVerification because:

• changing an interface of PublisherVerificationRule (although it is marked as internal) is not compatible,
• changing the PublisherVerification surface is not binary compatible and
• may result in bitterness as new failures require more work to be fixed in a library previously deemed verified.

[viktorklang]

@akarnokd Would it be possible to select a verification based on the length of the Publisher being tested?

[akarnokd]

The problem with multiple sub-cases inside an outer @Test case is that if one fails, the rest may not run, plus if one of them is optional due to the length (for example, checking §3.9 on an empty Publisher that may or may not ignore requests), that may still trigger skipping other sub-cases. I don't know if TestNG allows features such as JUnit 5 where you can have multiple sub-cases and run them all within the same test method.

[viktorklang]

@akarnokd Wouldn't it be possible to skip the tests which are not applicable by making them pass?

[akarnokd]

Currently, if a test is not applicable the implementation throws:

      throw new SkipException(String.format("Unable to run this test, as required elements nr: %d is higher than supported by given producer: %d", elements, maxElementsFromPublisher()));

We'd have to collect all such exceptions and rethrow a composite of them, i.e., SkipException if sub-tests were skipped or AssertionError if some tests failed + some were skipped.

The question is, what style of changes are you (and @ktoso ) willing to accept to the test classes.

[viktorklang]

My personal preference are: better TCK coverage, minimal-if-any-BC-breakages, better TCK failure error messages.

@ktoso?