Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"description" not printing when verify args don't match #1712

Closed
tir38 opened this issue May 18, 2019 · 8 comments
Closed

"description" not printing when verify args don't match #1712

tir38 opened this issue May 18, 2019 · 8 comments

Comments

@tir38
Copy link

tir38 commented May 18, 2019

Mockito version 2.27.0

I have a test that uses description inside a verify method. The description is correctly printing when the test fails because we don't interact with the mock

import static org.mockito.Mockito.description;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;

import org.junit.Test;

public class ExampleTest {
    @Test
    public void shouldFailWithDescription() {
        Dependency dependency = spy(Dependency.class);
        SystemUnderTest systemUnderTest = new SystemUnderTest();
        systemUnderTest.doSomething(dependency);
        verify(dependency, description("Failed to call doSomethingElse")).doSomethingElse(false);
    }

    static class SystemUnderTest {
        void doSomething(Dependency dependency) {
            // dependency.doSomethingElse(true); // <---- TEST FAILS BECAUSE WE DON'T INTERACT WITH MOCK
        }
    }

    static class Dependency {
        void doSomethingElse(boolean value) {
            // ...
        }
    }
}

Results in this failure message:

org.mockito.exceptions.base.MockitoAssertionError: Failed to call doSomethingElse <---- SEE OUR DESCRIPTION

Wanted but not invoked:
dependency.doSomethingElse(false);
-> at com.example.ExampleTest$Dependency.doSomethingElse(ExampleTest.java:28)
Actually, there were zero interactions with this mock.

In this case the test failed because there were no interactions with this mock. However if we update the SUT so that it calls the method with the wrong argument:

public class ExampleTest {
    @Test
    public void shouldFailWithDescription() {
        Dependency dependency = spy(Dependency.class);
        SystemUnderTest systemUnderTest = new SystemUnderTest();
        systemUnderTest.doSomething(dependency);
        verify(dependency, description("Failed to call doSomethingElse")).doSomethingElse(false);
    }

    static class SystemUnderTest {
        void doSomething(Dependency dependency) {
             dependency.doSomethingElse(true);  // <---- TEST FAILS BECAUSE WE PASS WRONG ARGS
        }
    }

    static class Dependency {
        void doSomethingElse(boolean value) {
            // ...
        }
    }
}

The test still fails. But this time it does not include the description:

Argument(s) are different! Wanted: <----- WE SHOULD SEE DESCRIPTION
dependency.doSomethingElse(false);
-> at com.example.ExampleTest$Dependency.doSomethingElse(ExampleTest.java:28)
Actual invocation has different arguments:
dependency.doSomethingElse(true);
-> at com.example.ExampleTest$SystemUnderTest.doSomething(ExampleTest.java:21)

Comparison Failure: 
Expected :dependency.doSomethingElse(false);
Actual   :dependency.doSomethingElse(true);
@tir38 tir38 closed this as completed May 18, 2019
@tir38 tir38 reopened this May 18, 2019
@tir38
Copy link
Author

tir38 commented May 18, 2019

Not pretty but this seems to work if I use a VerificationListener and manually log the error:

import static org.mockito.Mockito.description;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;

import org.junit.Test;
import org.mockito.Mockito;
import org.mockito.MockitoFramework;
import org.mockito.listeners.VerificationListener;

public class ExampleTest {
    @Test
    public void shouldFailWithDescription() {
        MockitoFramework framework = Mockito.framework();
        String failureMessage = "Failed to call doSomethingElse";

        // not all test failures will print the description so we have to manually log it =(
        VerificationListener listener = verificationEvent -> {
            if (verificationEvent.getVerificationError() != null) {
                System.out.println("FAILURE: " + failureMessage);
            }
        };
        framework.addListener(listener);

        Dependency dependency = spy(Dependency.class);
        SystemUnderTest systemUnderTest = new SystemUnderTest();
        systemUnderTest.doSomething(dependency);
        verify(dependency, description(failureMessage)).doSomethingElse(false);

        // not sure if we need to remove for next test, or if mockito will auto-clear
        framework.removeListener(listener);
    }

    static class SystemUnderTest {
        void doSomething(Dependency dependency) {
            dependency.doSomethingElse(true);  // <---- TEST FAILS BECAUSE WE PASS WRONG ARGS
        }
    }

    static class Dependency {
        void doSomethingElse(boolean value) {
            // ...
        }
    }
}

now prints out

FAILURE: Failed to call doSomethingElse

Argument(s) are different! Wanted:
dependency.doSomethingElse(false);
-> at com.example.ExampleTest$Dependency.doSomethingElse(ExampleTest.java:47)
Actual invocation has different arguments:
dependency.doSomethingElse(true);
-> at com.example.ExampleTest$SystemUnderTest.doSomething(ExampleTest.java:40)

Comparison Failure: 
Expected :dependency.doSomethingElse(false);
Actual   :dependency.doSomethingElse(true);

@jjDevPL
Copy link

jjDevPL commented May 28, 2019

I would like to analyze this issue and eventually provide fixes.

@tasomaniac
Copy link

I can also verify the same issue.

@mockitoguy
Copy link
Member

Seems like a bug. Please submit a PR. Thanks!

@adrianriley
Copy link
Contributor

adrianriley commented Jun 15, 2020

The cause of this is that Reporter.argumentsAreDifferent() can return

  • an org.mockito.exceptions.verification.opentest4j.ArgumentsAreDifferent, a subclass of org.opentest4j.AssertionFailedError
  • an org.mockito.exceptions.verification.junit.ArgumentsAreDifferent, a subclass of junit.framework.ComparisonFailure

according to whether opentest or junit is available in the classpath. Neither of these, crucially, extends MockitoAssertionError. Hence the exception is not caught in Description.verify and wrapped in a MockitoAssertionError with the description. That would only happen if neither opentest nor junit were in the classpath.

I think the catch block there should also catch these other exceptions and handle them in a similar way to include the description text.

@tasomaniac
Copy link

It's extending those because of tooling and IDE support, I believe. In IntelliJ IDEA for example, this error is displayed really nice. You can see and compare the differences on the arguments.

@adrianriley
Copy link
Contributor

The various ArgumentsAreDifferent exceptions come from org.mockito.internal.junit.ExceptionFactory, which deals solely with that condition. The header comment of MockitoAssertionError suggests it should be the base of all verification errors triggered by verify() methods. So maybe some refactoring is required.

I don't know enough to say whether the solution is to handle the exceptions which are actually being thrown or change to throw subclasses of MockitoAssertionError which would then fix the issue.

@adrianriley
Copy link
Contributor

Created a PR with a possible solution: #1949

epeee pushed a commit to epeee/mockito that referenced this issue Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants