JIT: RBO's utilization of liberal VNs is illegal

[jakobbotsch]

The following program hits the assert when run with DOTNET_JitNoCSE=1.

using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Threading;

public unsafe class Program
{
    public static void Main()
    {
        C c = new C();
        new Thread(_ => { while (true) { c.X = 0; c.X = 1; } }) { IsBackground = true }.Start();

        while (true)
        {
            IllegalRBO(c, 0);
        }
    }

    public class C
    {
        public int X;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void IllegalRBO(C c, int k)
    {
        int y = c.X;

        if (c.X == k + 1)
        {
            if (y == k + 1)
            {
                if (Id(y) != k + 1)
                {
                    Trace.Fail("Impossible");
                }
            }
        }
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Id(int x) => x;
}

RBO eliminates the y == k + 1 branch because it is dominated by c.X == k + 1 with the same liberal VN. However, this is not legal since y and c.X evaluated in the dominating test may not have the same value under data races.

The example requires DOTNET_JitNoCSE because otherwise CSE removes one of the loads of c.X, but naturally CSE could decide not to CSE these for any number of reasons (like register pressure).

cc @dotnet/jit-contrib

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.