Android Java 8 `java.time.Duration` postmortem
Mockito 3 is compatible with Java 8, compared to Java 6 for Mockito 2.
The Android ecosystem supports the Java 8 runtime, but does not support all new methods introduced in Java 8.
https://github.com/mockito/mockito/pull/1818 introduced a new API that relies on java.time.Duration, which does not exist on Android SDK's before 26.
If an Android user has 1 test that has to run on an SDK < 26, they would be broken by Mockito 3.2.0.
This constituted a breaking change.
The PR in question was reverted in https://github.com/mockito/mockito/pull/1845 and published as Mockito 3.2.4 to Maven Central.
For users who started using the new API introduced in Mockito 3.2.0 would be broken by Mockito 3.2.4.
This also constituted a breaking change.
Overall, there are no breaking changes comparing Mockito 3.1.0 and 3.2.4, but there were breaking changes comparing Mockito 3.1.0 and 3.2.0 (for Android users) and 3.2.0 and 3.2.2 (for users of the new API).
Thus far, 1 user reported they were using the new API and were broken by this change (https://github.com/mockito/mockito/issues/1843#issuecomment-567235571) and 1 user reported they were broken by the reference to java.time.Duration (https://github.com/mockito/mockito/issues/1843#issue-538087263).
The number of devices running SDK < 26 is ~60% (https://developer.android.com/about/dashboards). Mockito 3.2.0 has 190 reported usages on Maven Central (https://mvnrepository.com/artifact/org.mockito/mockito-core). Mockito 3.1.0 has 562 reported usages and Mockito 3.2.4 has 9 reported usages. For additional perspective: Mockito 3.0.0 has 954 reported usages, Mockito 1.10.19 has 5277 reported usages and the most commonly used Mockito 2 artifact is Mockito 2.23.4 with 2072 reported usages.
Our Continuous deployment model allowed us to upload a newer version of Mockito in a relatively short amount of time. There was no need for waiting for authentication from a Maven Central owner or Mockito owner to allow for publication. This lead to a publication time to Maven Central of <1 hour.
Mockito is used in both the Java and Android ecosystems. While we do provide a compatibility promise for the Java ecosystem (via our "minimum Java version" policy), we do not have such a policy in place for our Android users. Currently Android users expect their tests to run if they can use Java 8 themselves. However, Android users require desugaring (https://jakewharton.com/d8-library-desugaring/), which users appear to be unaware of.
We were unaware of the breakage and published to Maven Central, resulting in a user report 16 days after we published. Adding small Android smoke tests could prevent downstream breakages that are caught late in the cycle.
While the first change was a breaking change, the fix itself constituted a breaking change as well. We have no decision making process how to resolve such a situation. In this particular scenario, reverting was deemed the least intrusive option. We have no qualitative measurement on what "least intrusive" means.
We should update our documentation to include guidance on how to apply desugaring and point users to resolution. This will allow us to make use of types that are desugarable.
This is an unfortunate situation for Mockito to be in, but given our large Android userbase, it would be bad for Mockito's future to break these users. If a type can't be desugared, we should not be using it yet, to allow our users to remain using Mockito.
The previous point limits the evolution of Mockito, which is not a situation we want to be in. If we disocver that a type is not supported via desugaring yet, we should reach out to the Android toolchain owners to resolve the situation.
Right now, the number of devices running SDK < 26 is 60%. We should decide at which point we stop supporting this use case. Tentative proposal: less than 25% marketshare would be a good threshold. We should discuss the exact number and make this clear in our documentation.