Should().BeEquivalentTo() show different error message between Debug and Release

[souhailelk]

Description

We compile and launch unit test libraries with the release configuration and x64 platform.
After upgrading FluentAssertions from 4.19.2 to 6.5.1 (the latest), we noticed that Should().BeEquivalentTo() gives different error messages while compare classes with properties.
Let's look at the following example:

• Let ClassWithProperty be a class containing one property IntProperty:

public sealed class ClassWithProperty
{
    public ClassWithProperty(int value)
    {
        IntProperty = value;
    }
    public int IntProperty { get; }
}

• The unit test CompareClassWithPropertyObjects compare two objects of type ClassWithProperty, with different property values.

[Test]
public void CompareClassWithPropertyObjects()
{
   //Arrange
    var actual = new ClassWithProperty(1);
    var expected = new ClassWithProperty(2);
    
    //Assert
    actual.Should().BeEquivalentTo(expected);
}

Expected behavior:

Expected property actual.IntProperty to be 2, but found 1.

With configuration:
- Use declared types and members
- Compare enums by value
- Compare tuples by their properties
- Compare anonymous types by their properties
- Compare records by their members
- Match member by name (or throw)
- Be strict about the order of items in byte arrays
- Without automatic conversion.

Actual behavior:

Here, it outputs the }}}, instead of the name of the actual object.

Expected property }}}.IntProperty to be 2, but found 1.

With configuration:
- Use declared types and members
- Compare enums by value
- Compare tuples by their properties
- Compare anonymous types by their properties
- Compare records by their members
- Match member by name (or throw)
- Be strict about the order of items in byte arrays
- Without automatic conversion.

Versions

• .NET : .NET Core 3.1 and .NET 4.7.1
• NUnit : 3.12.0 (I tried also the the version 3.13.1, It gives the same results).
• FluentAssertions : 6.5.1
• Configuration : Release
• Platform : x64

Note that it works totally fine in Debug configuration.

[dennisdoomen]

As is documented here, you need to run your tests in Debug mode for FA to be able to extract the variable from the stack trace. Otherwise the compiler will optimize those details away. However, it should fall back to using a generic name, so technically it's a little bug.

[jnyrup]

I did a related analysis in #1781 (comment)

[dennisdoomen]

So I guess this is fixed by #1788?

[jnyrup]

So I guess this is fixed by #1788?

No. That fix was released with 6.4.0, and this issue was reproduced with 6.5.1

The issue here is that }}} is more noisy/misleading than not determining the caller identity at all.
If DetermineCallerIdentity had returned null instead of }}}, the failure message becomes

Expected property root.IntProperty to be 2, but found 1.

using Xunit;

namespace FluentAssertions.Specs
{
    public class SomeSpecs
    {
        public sealed class ClassWithProperty
        {
            public ClassWithProperty(int value)
            {
                IntProperty = value;
            }

            public int IntProperty { get; }
        }

        [Fact]
        public void CompareClassWithPropertyObjects()
        {
            // Arrange
            var actual = new ClassWithProperty(1);
            var expected = new ClassWithProperty(2);

            // Assert
            actual.Should().BeEquivalentTo(expected);
        }
    }
}

[ghost]

Can this be rechecked? I tried this test in release mode and I do actually see root, not }}}.

[jnyrup]

@sdelarosbil which TFM are you using?
I just retried my example above in a fresh project targeting net472 and referencing FA 6.1.10 and I still see }}}.

[ghost]

@sdelarosbil which TFM are you using? I just retried my example above in a fresh project targeting net472 and referencing FA 6.1.10 and I still see }}}.

I tried on the develop branch using .net framework 4.7 and I still get root.

I am just wondering if it's been fixed since it was last checked on develop.

[jnyrup]

Here's my output with net472 running in release mode.

nuget reference to FA 6.1.10

project reference to develop

[ghost]

Ok, I got it, sorry about that, it seems if I add the test inside the existing ones, it doesn't reproduce, but if I create a project then references the develop one, now it reproduces.

[ghost]

This seems complex to address because the only reason it's putting }}} is because that's what was in the lines below due to the runtime info messing up the symbols accuracy (they are generated by default in release mode).

But you could have put another Should() call below that line which will now print something valid, but wrong such as if you had this instead:

        [Fact]
        public void CompareClassWithPropertyObjects()
        {
            // Arrange
            var actual = new ClassWithProperty(1);
            var expected = new ClassWithProperty(2);

            // Assert
            actual.Should().BeEquivalentTo(expected);
            expected.Should();
        }

This will print:

Expected property expected.IntProperty to be 2, but found 1.

So you can't easily rely on the fact it could not be a valid identifier because it could be valid, but completely wrong.

The funny thing is you can actually make it return null if you set the project to not generate the pdb in release mode. At least in Rider, I have the option to disable jit optimizations in release mode if the debugger starts the process which allows to get the true name too.

So the way I see it, I see 2 ways we could address this:

1. Find a way to detect the return of the call identifier being valid or not.
2. Leave this be and tell people that release mode symbols can trick our call identification to give false information.

Not sure which one is best here because the fact runtime jit optimizations can affect this makes this complicated to detect.

[dennisdoomen]

Or reintroduce the imperfect solution that disables caller identification if it detects a non-debug build?

[ghost]

Or reintroduce the imperfect solution that disables caller identification if it detects a non-debug build?

Either that or not compile in release mode with symbols since that forces the resolver to not see anything and correctly resolve to null.

[IT-VBFK]

I think I have found somewhat related to this?

In my case (Visual Studio, macOS) a few assertions (or tests) fails, because it cannot extract the subject name and therefore it falls back to the default value.

e.g. This test (note: it has nothing to with what the test is about, this works fine)

[Fact]
public void Mismatches_in_multiline_text_includes_the_line_number()
{
    var expectedIndex = 100 + (4 * Environment.NewLine.Length);

    var subject = """
    @startuml
    Alice -> Bob : Authentication Request
    Bob --> Alice : Authentication Response

    Alice -> Bob : Another authentication Request
    Alice <-- Bob : Another authentication Response
    @enduml
    """;

    var expected = """
    @startuml
    Alice -> Bob : Authentication Request
    Bob --> Alice : Authentication Response

    Alice -> Bob : Invalid authentication Request
    Alice <-- Bob : Another authentication Response
    @enduml
    """;

    // Act
    Action act = () => subject.Should().Be(expected);

    // Assert
    act.Should().Throw<XunitException>().WithMessage($"""
        Expected subject to be the same string, but they differ on line 5 and column 16 (index {expectedIndex}):
                     ↓ (actual)
          "…-> Bob : Another…"
          "…-> Bob : Invalid…"
                     ↑ (expected).
        """);
}

throws this failure on my mac (note the difference: string instead of subject):

Expected exception message to match the equivalent of "Expected subject to be the same string, but they differ on line 5 and column 16 (index 104):
             ↓ (actual)
  "…-> Bob : Another…"
  "…-> Bob : Invalid…"
             ↑ (expected).", but "Expected string to be the same string, but they differ on line 5 and column 16 (index 104):
             ↓ (actual)
  "…-> Bob : Another…"
  "…-> Bob : Invalid…"
             ↑ (expected)." does not.

[dennisdoomen]

I think I have found somewhat related to this?

Did you build your tests in debug mode?

[IT-VBFK]

Yes

[dennisdoomen]

Yes

I've just tried your example, but it works on my Windows machine

Expected subject to be the same string, but they differ on line 5 and column 16 (index 108):
             ↓ (actual)
  "…-> Bob : Another…"
  "…-> Bob : Invalid…"
             ↑ (expected).

Either way, this seems to be unrelated to the original issue. So please create a separate issue so to not obfuscate the original issue.