System.IO.Net5Compat.Tests and System.IO.Tests suddenly exiting with error 137

[carlossanlop]

The System.IO.Net5Compat.Tests and the System.IO.Tests test processes are intermittengly getting killed on Linux shortly after starting, and the exit code is 137.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407
Build error leg or test failing: System.IO.Net5Compat.Tests

Error Message

{
  "ErrorPattern": ["Starting:    System\\.IO\\.(Net5Compat\\.)?Tests", "exit code 137"],
  "BuildRetry" : true,
  "ExcludeConsoleLog" : false
}

System.IO.Net5Compat.Tests example

• PR: Update branding to 6.0.30 #100545
• Queue: Libraries Test Run release coreclr Linux_musl x64 Debug
• Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407&view=logs&j=73b7db51-6fbc-594d-25c9-15b2737b0bf6
• Log file: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-100545-merge-f07349d435f3464e8b/System.IO.Net5Compat.Tests/1/console.71f42a07.log?helixlogtype=result
• Output:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.IO.Net5Compat.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.Net5Compat.Tests (found 679 of 685 test cases)
  Starting:    System.IO.Net5Compat.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Net5Compat.Tests.runtimeconfig.json --depsfile System.IO.Net5Compat.Tests.deps.json xunit.console.dll System.IO.Net5Compat.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Apr 2 20:20:02 UTC 2024 ----- exit code 137 ----------------------------------------------------------

System.IO.Test example

• PR: Update branding to 6.0.30 #100545
• Queue: Libraries Test Run release mono Linux x64 Debug
• Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407&view=logs&j=b4344b0d-0f92-5d69-ccaf-e0b24fbf14a2&t=65a58faa-a542-5344-b5c8-897139f242cd&l=888
• Log file: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-100545-merge-f07349d435f3464e8b/System.IO.Tests/1/console.effe990e.log?helixlogtype=result
• Output:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.Tests (found 679 of 685 test cases)
  Starting:    System.IO.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Apr 2 20:20:10 UTC 2024 ----- exit code 137 ----------------------------------------------------------

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407
Error message validated: [Starting: System\.IO\.(Net5Compat\.)?Tests exit code 137]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/2/2024 11:08:28 PM UTC

Report

Build	Definition	Test	Pull Request
687947	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102558
687914	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102498
687812	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102695
687677	dotnet/runtime	System.IO.Tests.WorkItemExecution	#99183
687512	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102558
687437	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102685
687411	dotnet/runtime	System.IO.Tests.WorkItemExecution	#100951
687264	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102180
687041	dotnet/runtime	System.IO.Tests.WorkItemExecution
687027	dotnet/runtime	System.IO.Tests.WorkItemExecution	#99183
687009	dotnet/runtime	System.IO.Tests.WorkItemExecution	#100951
687005	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102364
686965	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102379
686943	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102592
686921	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102667
686906	dotnet/runtime	System.IO.Tests.WorkItemExecution
686264	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102640
686579	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102558
686376	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102484
686269	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102639
686134	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102604
686115	dotnet/runtime	System.IO.Tests.WorkItemExecution
686008	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102636
685888	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102624
685866	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102404
685587	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102509
685572	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102604
685469	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102605
685382	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102558
685238	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101701
685213	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101531
685210	dotnet/runtime	System.IO.Tests.WorkItemExecution	#100951
685183	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102547
684969	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102589
684966	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102588
684779	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102364
2458479	dotnet-runtime	System.IO.Tests.WorkItemExecution	#39678
684520	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102295
684439	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102499
684424	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102543
684369	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102262
684145	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102493
684125	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102558
682788	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102495
2457807	dotnet-runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#39678
683532	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101753
683498	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102529
682950	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101295
682820	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102497
682321	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102469
681993	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102096
682003	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution
682154	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102364
681981	dotnet/runtime	System.IO.Tests.WorkItemExecution
681893	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102402
681892	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102466
681949	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102461
681829	dotnet/runtime	System.IO.Tests.WorkItemExecution
681713	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102295
681699	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102464
681400	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101531
681371	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102188
681111	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102444
681073	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102366
680954	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102437
680933	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102436
680906	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102435
681018	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102188
680657	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102420
680622	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102188
680039	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102364
680028	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102180
679941	dotnet/runtime	System.IO.Tests.WorkItemExecution
679809	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102402
679808	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102401
679762	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102346
679754	dotnet/runtime	System.IO.Tests.WorkItemExecution
679742	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102399
679597	dotnet/runtime	System.IO.Tests.WorkItemExecution
679613	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102346
679557	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102364
2454087	dotnet-runtime	System.IO.Tests.WorkItemExecution	#39678
679308	dotnet/runtime	System.IO.Tests.WorkItemExecution
679269	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102301
679018	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102331
678702	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102363
678547	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102355
678666	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101701
678583	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101662
678371	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#102346
678358	dotnet/runtime	System.IO.Tests.WorkItemExecution
678265	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102298
678084	dotnet/runtime	System.IO.Tests.WorkItemExecution	#101947
678008	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102331
677889	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102295
2452732	dotnet-runtime	System.IO.Net5Compat.Tests.WorkItemExecution	#39614
677365	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102302
677242	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102270
676946	dotnet/runtime	System.IO.Tests.WorkItemExecution	#102139
676930	dotnet/runtime	System.IO.Tests.WorkItemExecution
Displaying 100 of 177 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	68	177

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[ericstj]

@dotnet/area-system-io there are a lot of hits on this and relatively recent. It seems to me to be happening across many configurations. I think it's worth having a look.

[adamsitnik]

The System.IO.Net5Compat.Tests and the System.IO.Tests test processes are intermittengly getting killed on Linux shortly after starting, and the exit code is 137.

137 means out of memory. We have not made any changes to 6.0 in System.IO, so I expect that either there was some infra change (like less memory available) or a bug was introduced in the product itself. The bug would be specific to Linux.

@carlossanlop is it possible to perform some kind of binary search based on the merged PRs and when it started to fail?