Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runaway Node process consuming CPU #1113

Open
t0yv0 opened this issue Oct 23, 2023 · 2 comments
Open

Runaway Node process consuming CPU #1113

t0yv0 opened this issue Oct 23, 2023 · 2 comments
Labels
impact/performance Something is slower than expected impact/reliability Something that feels unreliable or flaky kind/bug Some behavior is incorrect or out of spec

Comments

@t0yv0
Copy link
Member

t0yv0 commented Oct 23, 2023

What happened?

When running example tests in this repo I noticed 3 very long-running node processes consuming CPU on my Mac machine. This seems to be Pulumi Node language hosts as far as I can tell and they're orphan processes in the sense that their process group has terminated. I think something must be wrong with self-destruct/heartbeat machinery. Not filing it to pulumi/pulumi yet as this provider is special as the provider itself is written in Node and possibly is the culprit here.

ps -eaf | grep 76892                                                                                                                                                                        ~/code/pulumi-awsx/examples/ecs/nodejs
  502 76892     1   0 Fri01PM ??       364:50.85 /Users/t0yv0/bin/node /private/var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/p-it-antons-mac-ts-lb-simp-6b81ed54-1927854174/node_modules/@pulumi/pulumi/cmd/run/index.js --monitor 127.0.0.1:59921 --engine 127.0.0.1:59916 --sync /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/pulumi-node-pipes188323378 --organization organization --project ts-lb-simple --stack p-it-antons-mac-ts-lb-simp-6b81ed54 --pwd /private/var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/p-it-antons-mac-ts-lb-simp-6b81ed54-1927854174 --query-mode false --parallel 2147483647 .
  502  4423  4217   0  2:46PM ttys002    0:00.00 grep 76892

Example

N/A

Output of pulumi about

v3.86.0.

Additional context

N/A

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@t0yv0 t0yv0 added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Oct 23, 2023
@mikhailshilkov
Copy link
Member

Do you have a theory about how this provider can be the cause of this, as opposed to a Core issue? Node providers are a supported scenario, so it's not wrong per se.

@mikhailshilkov mikhailshilkov added impact/reliability Something that feels unreliable or flaky impact/performance Something is slower than expected and removed needs-triage Needs attention from the triage team labels Oct 26, 2023
@t0yv0
Copy link
Member Author

t0yv0 commented Oct 26, 2023

This is going to be tricky without a solid repro. I've not worked with the internals here much yet.

There's two issues:

  • why is it spinning CPU, what is it doing
  • why does it not self-destruct after the connection to the engine is lost

For the second issue we may have failed to implement heartbeat self-destruct for Node (std feature of Go providers).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/performance Something is slower than expected impact/reliability Something that feels unreliable or flaky kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Development

No branches or pull requests

2 participants