Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI out of memory conditions on Windows #263

Merged
merged 5 commits into from Apr 20, 2022

Conversation

AaronFriel
Copy link
Member

@AaronFriel AaronFriel commented Apr 19, 2022

Templates fail to run on Windows, encountering an error indicative of OOM.

Workaround: Reduce CI test parallelism on Windows to 3 from 5.

Unrelated change and first attempt at fix (first commit) moved more cleanup actions into the t.Cleanup() step, ensuring they are ordered by the Go test harness and subtests are allowed to run their cleanups respectively on test failure. This should have the side effect of reducing orphan resources in our cloud accounts, assuming the cleanup step is allowed to run.

Root cause analysis

The last successful run: https://github.com/pulumi/templates/runs/5804552689?check_suite_focus=true all green with

  • CLI v3.28.0
  • Dev CLI v3.29.0-alpha.1648883111

First failing run began failing on the dev CLI: https://github.com/pulumi/templates/runs/5812628677?check_suite_focus=true

  • CLI v3.28.0
  • Dev CLI 3.29.0-alpha.1648997699

As the commit hash for the repo hadn't changed, only the Pulumi CLI, I looked for changes there between those two timestamps:

git log --since=1648883110 --until=1648997700 --oneline -m
1d215c2c0 Move InstallDependencies to the language plugin (#9294)
2151ab6e1 Fix python protobuf to < 3.20.0 (#9336)

There was only one relevant merge commit in that time frame, referencing this PR pulumi/pulumi#9294 which modified pulumi new to invoke language plugins, which would mean spawning additional processes. Additional processes, greater memory burden, and the raw logs point to the same error we've seen on pulumi/pulumi when we've hit out of memory conditions:

2022-04-04T08:16:26.6485820Z ##[error]    template_test.go:88: Starting test run for "gcp-visualbasic"
2022-04-04T08:16:26.6488687Z ##[error]    environment.go:87: Created new test environment:  C:\Users\RUNNER~1\AppData\Local\Temp\test-env475742893
2022-04-04T08:16:26.6491081Z ##[error]    template_test.go:107: Running command pulumi new gcp-visualbasic -f --yes -s template-test
2022-04-04T08:16:26.6493089Z ##[error]    template_test.go:107: Ran command pulumi args [new gcp-visualbasic -f --yes -s template-test] and expected success. Instead got failure.
2022-04-04T08:16:26.6494618Z ##[error]    template_test.go:107: Run Error: exit status 2
2022-04-04T08:16:26.6496128Z ##[error]    template_test.go:107: STDOUT: 
2022-04-04T08:16:26.6498045Z ##[error]    template_test.go:107: STDERR: panic: Failed to load user32.dll: A dynamic link library (DLL) initialization routine failed.
2022-04-04T08:16:26.6499222Z         
2022-04-04T08:16:26.6499933Z         goroutine 1 [running]:
2022-04-04T08:16:26.6500489Z         syscall.MustLoadDLL(...)
2022-04-04T08:16:26.6501351Z         	/Users/runner/hostedtoolcache/go/1.17.8/x64/src/syscall/dll_windows.go:106
2022-04-04T08:16:26.6511222Z         github.com/xanzy/ssh-agent.winAPI(***0x1ef6ee0, 0x2fbd950***, ***0x1ef78c2, 0xb***)
2022-04-04T08:16:26.6512600Z         	/Users/runner/go/pkg/mod/github.com/xanzy/ssh-agent@v0.2.1/pageant_windows.go:68 +0xb4
2022-04-04T08:16:26.6513395Z         github.com/xanzy/ssh-agent.init()
2022-04-04T08:16:26.6514206Z         	/Users/runner/go/pkg/mod/github.com/xanzy/ssh-agent@v0.2.1/pageant_windows.go:62 +0x1ac
2022-04-04T08:16:26.6514857Z     template_test.go:111: 
2022-04-04T08:16:26.6515433Z         	Error Trace:	template_test.go:111
2022-04-04T08:16:26.6516242Z         	Error:      	Should NOT be empty, but was 
2022-04-04T08:16:26.6516979Z         	Test:       	TestTemplates/gcp-visualbasic
2022-04-04T08:16:26.6517922Z panic: fatal: A precondition has failed for path [recovered]
2022-04-04T08:16:26.6518640Z 	panic: fatal: A precondition has failed for path

@AaronFriel AaronFriel changed the title fix tests on Windows by deferring deletions until after all subtests have completed Fix CI out of memory conditions on Windows Apr 19, 2022
@AaronFriel AaronFriel merged commit 08b6936 into master Apr 20, 2022
@pulumi-bot pulumi-bot deleted the aaronfriel/windows-tests branch April 20, 2022 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant