secnetperf TCP Cleanup Synchronization Issues

[nibanks]

Describe the bug

A crash has occurred in our secnetperf automation during the TCP HPS tests.

0b 000000a8`b53fecc0 00007ff9`4d39749d     ntdll!KiUserExceptionDispatcher+0x2e
0c 000000a8`b53ff3f0 00007ff6`aeaf760f     ntdll!RtlEnterCriticalSection+0xd
0d 000000a8`b53ff420 00007ff6`aeb0adfb     secnetperf!TcpConnection::ReceiveCallback+0xaf [D:\a\netperf\netperf\src\perf\lib\Tcp.cpp @ 489] 
0e 000000a8`b53ff520 00007ff6`aeb0afe4     secnetperf!CxPlatDataPathTcpRecvComplete+0x15b [D:\a\netperf\netperf\src\platform\datapath_winuser.c @ 3433] 
0f (Inline Function) --------`--------     secnetperf!CxPlatDataPathRecvComplete+0x2c [D:\a\netperf\netperf\src\platform\datapath_winuser.c @ 3521] 
10 000000a8`b53ff5f0 00007ff6`aeb0be2e     secnetperf!CxPlatDataPathSocketProcessReceive+0xc4 [D:\a\netperf\netperf\src\platform\datapath_winuser.c @ 3561] 
11 000000a8`b53ff650 00007ff6`aeb0582e     secnetperf!DataPathProcessCqe+0x17e [D:\a\netperf\netperf\src\platform\datapath_winuser.c @ 4368] 
12 (Inline Function) --------`--------     secnetperf!CxPlatDataPathProcessCqe+0x25 [D:\a\netperf\netperf\src\platform\datapath_win.c @ 722] 
13 (Inline Function) --------`--------     secnetperf!CxPlatProcessEvents+0xb1 [D:\a\netperf\netperf\src\platform\platform_worker.c @ 454] 
14 000000a8`b53ff680 00007ff9`4be04de0     secnetperf!CxPlatWorkerThread+0x22e [D:\a\netperf\netperf\src\platform\platform_worker.c @ 497] 
15 000000a8`b53ff9d0 00007ff9`4d3fedab     kernel32!BaseThreadInitThunk+0x10
16 000000a8`b53ffa00 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

I took a quick look and this means that somehow TCP socket layer must be calling up for a receive indication even though the TcpConnection has already been deleted. But in the destruction it clearly calls CxPlatSocketDelete. Looking at the SocketDelete code path, it seems to have some special cases for UseTcp and I think it shouldn't. My guess here is that this is what results in sometimes TCP code not waiting for all cleanup to occur before cleanup

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

Run netperf perf tests

Expected behavior

All succeed

Actual outcome

TCP regularly crashes or hangs.

Additional details

netpef run: https://github.com/microsoft/netperf/actions/runs/7596674678
Dump and symbols: https://github.com/microsoft/netperf/actions/runs/7596674678/artifacts/1183278444

• Note to look at the secnetperf.exe.4376.dmp dump (the other one is a hang, but I suspect this is the underlying cause)