Details
-
Test
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
We've started using strict stubbing for unit tests written with Mockito, which is supposed to automatically fail tests when they set up mock expectations that go unused.
However, these failures are not reported during Jenkins builds, even if they are reported when building/testing locally.
In at least one case, this difference appears to be because our Jenkins build uses the custom unitTest and integrationTest tasks defined in the project's Gradle build file, instead of the test task. Some IDEs (such as IntelliJ) may use the latter instead of the former, which can cause tests to fail due to unnecessary stubbings when being run in that IDE but not when being built on Jenkins.
It's possible that, because the custom test tasks filter out some tests from running, Mockito does not check for unnecessary stubbings in order to avoid incorrectly failing tests that set up mocks in, e.g., a @BeforeEach method.
This exact behavior has been reported elsewhere as a Gradle issue; based on discussion on that thread, it appears this is a known and somewhat-intentional limitation of Mockito:
I spent some time trying to solve this and eventually I stumbled upon this piece in Mockito's JUnit runner:
// only report when:
// 1. if all tests from given test have ran (filter requested is false)
// Otherwise we would report unnecessary stubs even if the user runs just single test
// from the class
// 2. tests are successful (we don't want to add an extra failure on top of any existing
// failure, to avoid confusion)
(1) suggests that skipping unused stub validation is the intended behavior when the user filters a single test from the class. However, this behavior applies to any type of filter.And Gradle indeed applies a CategoryFilter if categories are configured: https://github.com/rieske/gradle/blob/e82029abb559d620fdfecb4708a95c6313af625c/subprojects/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecutor.java#L70-L96
Which then causes Mockito to not validate unused stubs.