secnetperf not working for big payload sizes

[PratikKumarSaha]

Describe the bug

The QUIC version (till 7th Oct 2023) is working fine on my machine for payload size of 1.5GB, however, it is not working for the commits after 7th October 2023.

We have assigned 2 CPUs each for SERVER and CLIENT. For small payload size (eg : 150000 ) it works fine but not for bigger payload sizes. Following is the output to distinguish the results.

[root@9030bf879c51 x86_Debug_openssl]# ./secnetperf -exec:maxtput -test:tput -target:192.168.96.3 -up:150000 -ptput:1
Started!

Result: Upload 150000 bytes @ 236173 kbps (5.081 ms) .

[root@9030bf879c51 x86_Debug_openssl]# ./secnetperf -exec:maxtput -test:tput -target:192.168.96.3 -up:1500000000 -ptput:1
Started!

Aborted (core dumped) 

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

• NAME = "Red Hat Enterprise Linux"
• VERSION="8.8 (Ootpa)"
• ID = " rhel"
• Architecture: x86_64
• CPU op-mode(s): 32-bit, 64-bit
• Byte Order: Little Endian
• CPU(s) : 40
• On-line CPUS(s) list: 0-39
• Thread(s) per core: 2
• Core(s) per socket: 10
• Socket(s) : 2
• NUMA mode: 2
• Vendor ID: GenuineIntel
• CPU family: 6
• Model: 85
• Model Name: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz

MsQuic version

main

Steps taken to reproduce bug

1. /secnetperf -exec:maxtput (Server)
2. ./secnetperf -exec:maxtput -test:tput -target:192.168.96.3 -up:1500000000 -ptput:1 (Client)

Expected behavior

It should not have thrown core dumped since it was working properly for the commits before and om 7th October 2023.

Actual outcome

[root@9030bf879c51 x86_Debug_openssl]# ./secnetperf -exec:maxtput -test:tput -target:192.168.96.3 -up:150000 -ptput:1
Started!

Result: Upload 150000 bytes @ 236173 kbps (5.081 ms) .

[root@9030bf879c51 x86_Debug_openssl]# ./secnetperf -exec:maxtput -test:tput -target:192.168.96.3 -up:1500000000 -ptput:1
Started!

Aborted (core dumped) 

Additional details

No response

[nibanks]

Our output indicates a core dump. Would you mind sharing that and ideally the thread/callstack that's crashing?

[PratikKumarSaha]

#0  0x00007f3c88630acf in raise () from /lib64/libc.so.6
#1  0x00007f3c88603ea5 in abort () from /lib64/libc.so.6
#2  0x00007f3c89ba0918 in quic_bugcheck (File=0x7f3c89d4d088 "/msquic/src/platform/datapath_epoll.c", Line=2179, 
    Expr=0x7f3c89d4d5a8 "SendData->TotalSize + MaxBufferLength <= sizeof(SendData->Buffer)") at /msquic/src/platform/platform_posix.c:93
#3  0x00007f3c89ba92e8 in SendDataAllocBuffer (SendData=0x7f3c38017840, MaxBufferLength=<optimized out>) at /msquic/src/platform/datapath_epoll.c:2188
#4  0x00007f3c89ba25b8 in CxPlatSendDataAllocBuffer (SendData=<optimized out>, MaxBufferLength=<optimized out>) at /msquic/src/platform/datapath_linux.c:369
#5  0x00007f3c89b9c651 in QuicPacketBuilderPrepare (Builder=0x7f3c5f58ea40, NewPacketKeyType=QUIC_PACKET_KEY_1_RTT, IsTailLossProbe=<optimized out>, 
    IsPathMtuDiscovery=IsPathMtuDiscovery@entry=0 '\000') at /msquic/src/core/packet_builder.c:291
#6  0x00007f3c89b9cfb0 in QuicPacketBuilderPrepareForStreamFrames (Builder=Builder@entry=0x7f3c5f58ea40, IsTailLossProbe=<optimized out>)
    at /msquic/src/core/packet_builder.c:623
#7  0x00007f3c89b9283b in QuicSendFlush (Send=Send@entry=0x7f3c440021a0) at /msquic/src/core/send.c:1352
#8  0x00007f3c89b78445 in QuicConnDrainOperations (Connection=Connection@entry=0x7f3c440013e0) at /msquic/src/core/connection.c:7595
#9  0x00007f3c89b61953 in QuicWorkerProcessConnection (Worker=Worker@entry=0x1d6f710, Connection=0x7f3c440013e0, ThreadID=13507, 
    TimeNow=TimeNow@entry=0x7f3c5f58eeb0) at /msquic/src/core/worker.c:505
#10 0x00007f3c89b6222b in QuicWorkerLoop (Context=Context@entry=0x1d6f710, State=State@entry=0x7f3c5f58eeb0) at /msquic/src/core/worker.c:658
#11 0x00007f3c89b62626 in QuicWorkerThread (Context=0x1d6f710) at /msquic/src/core/worker.c:726
#12 0x00007f3c889af1ca in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3c8861be73 in clone () from /lib64/libc.so.6

[PratikKumarSaha]

Hello @nibanks any update on this issue?

[nibanks]

Sorry, we still haven't had a chance to take a look. This is not a known issue.

[PratikKumarSaha]

But this issue is not seen on the code version available till 7th Oct 2023. Are your platforms working for big payload sizes (1500000000) ?

[nibanks]

I haven't seen any issues with big payload. Have you tried Release builds? Perhaps that assert is benign.

[PratikKumarSaha]

Hello @nibanks I have tested the same for release v.2.3.4 and v.2.3.5 same issue is being observed for big payload sizes. Please provide us some insight on this.

[nibanks]

I mean, have you tried Release (not Debug) builds?

[PratikKumarSaha]

Yes , I have tried the releases both v.2.3.4 and v.2.3.5
Big payload sizes are failing for RedHat 8.8 environment

[nibanks]

I am not referring to a particular release branch or tag, but that you aren't building for debug mode. Asserts are only compiled in for debug mode, and I think the assert you're hitting is benign. If you are compiling/running release mode, please share the callstack for what is crashing now.

[PratikKumarSaha]

#0  0x00007f3c88630acf in raise () from /lib64/libc.so.6
#1  0x00007f3c88603ea5 in abort () from /lib64/libc.so.6
#2  0x00007f3c89ba0918 in quic_bugcheck (File=0x7f3c89d4d088 "/msquic/src/platform/datapath_epoll.c", Line=2179, 
    Expr=0x7f3c89d4d5a8 "SendData->TotalSize + MaxBufferLength <= sizeof(SendData->Buffer)") at /msquic/src/platform/platform_posix.c:93
#3  0x00007f3c89ba92e8 in SendDataAllocBuffer (SendData=0x7f3c38017840, MaxBufferLength=<optimized out>) at /msquic/src/platform/datapath_epoll.c:2188
#4  0x00007f3c89ba25b8 in CxPlatSendDataAllocBuffer (SendData=<optimized out>, MaxBufferLength=<optimized out>) at /msquic/src/platform/datapath_linux.c:369
#5  0x00007f3c89b9c651 in QuicPacketBuilderPrepare (Builder=0x7f3c5f58ea40, NewPacketKeyType=QUIC_PACKET_KEY_1_RTT, IsTailLossProbe=<optimized out>, 
    IsPathMtuDiscovery=IsPathMtuDiscovery@entry=0 '\000') at /msquic/src/core/packet_builder.c:291
#6  0x00007f3c89b9cfb0 in QuicPacketBuilderPrepareForStreamFrames (Builder=Builder@entry=0x7f3c5f58ea40, IsTailLossProbe=<optimized out>)
    at /msquic/src/core/packet_builder.c:623
#7  0x00007f3c89b9283b in QuicSendFlush (Send=Send@entry=0x7f3c440021a0) at /msquic/src/core/send.c:1352
#8  0x00007f3c89b78445 in QuicConnDrainOperations (Connection=Connection@entry=0x7f3c440013e0) at /msquic/src/core/connection.c:7595
#9  0x00007f3c89b61953 in QuicWorkerProcessConnection (Worker=Worker@entry=0x1d6f710, Connection=0x7f3c440013e0, ThreadID=13507, 
    TimeNow=TimeNow@entry=0x7f3c5f58eeb0) at /msquic/src/core/worker.c:505
#10 0x00007f3c89b6222b in QuicWorkerLoop (Context=Context@entry=0x1d6f710, State=State@entry=0x7f3c5f58eeb0) at /msquic/src/core/worker.c:658
#11 0x00007f3c89b62626 in QuicWorkerThread (Context=0x1d6f710) at /msquic/src/core/worker.c:726
#12 0x00007f3c889af1ca in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3c8861be73 in clone () from /lib64/libc.so.6

Hello @nibanks , I have verified the current release (2.3.5) and found the core dump stack is same as earlier which I have shared. Also, we have compiled it using "-Config Debug" and "-Config Release" modes. Both scenarios are having the same core dump call stack.

[nibanks]

SendDataAllocBuffer only has debug asserts (CXPLAT_DBG_ASSERT), so if you're hitting an assert, you must not actually be compiling or running the Release config:

_IRQL_requires_max_(DISPATCH_LEVEL)
_Success_(return != NULL)
QUIC_BUFFER*
SendDataAllocBuffer(
    _In_ CXPLAT_SEND_DATA* SendData,
    _In_ uint16_t MaxBufferLength
    )
{
    CXPLAT_DBG_ASSERT(SendData != NULL);
    CXPLAT_DBG_ASSERT(MaxBufferLength > 0);
    CxPlatSendDataFinalizeSendBuffer(SendData);
    CXPLAT_DBG_ASSERT(SendData->SegmentSize == 0 || SendData->SegmentSize >= MaxBufferLength);
    CXPLAT_DBG_ASSERT(SendData->TotalSize + MaxBufferLength <= sizeof(SendData->Buffer));
    CXPLAT_DBG_ASSERT(
        SendData->SegmentationSupported ||
        SendData->BufferCount < SendData->SocketContext->DatapathPartition->Datapath->SendIoVecCount);
    UNREFERENCED_PARAMETER(MaxBufferLength);
    if (SendData->ClientBuffer.Buffer == NULL) {
        return NULL;
    }
    SendData->ClientBuffer.Length = MaxBufferLength;
    return &SendData->ClientBuffer;
}

[PratikKumarSaha]

But on running in -Config Debug mode, I'm getting the same core dump call stack

[nibanks]

It can't be the same callstack on Release though. What is the callstack on Release?