Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SDK/Python] Parallelism Flag Not Respected #11116

Closed
Tracked by #11598
RobbieMcKinstry opened this issue Oct 21, 2022 · 0 comments · Fixed by #11122
Closed
Tracked by #11598

[SDK/Python] Parallelism Flag Not Respected #11116

RobbieMcKinstry opened this issue Oct 21, 2022 · 0 comments · Fixed by #11122
Assignees
@RobbieMcKinstry
Labels
area/language-host Runtime that executes user programs impact/performance Something is slower than expected kind/bug Some behavior is incorrect or out of spec language/python resolution/fixed This issue was fixed size/S Estimated effort to complete (1-2 days).
Milestone
0.79

Comments

@RobbieMcKinstry
Copy link
Contributor

What happened?

Python programs do not respect the --parallel flag. While the --parallel flag is plumbed through to the language host, and even into the the SDK at pulumi/runtime/settings.py, the value is never used.

Certain rpc calls generate blocking futures that the runtime offloads to a worker thread. For example, any use of run_in_executor throws the future into a worker thread. (Here's where we make that call.)

The default executor sets the max number of worker threads using a fixed value scaling based on the number of CPUs on the machine. This is the value that's currently employed.

This issue is to replace the default executor with one that respects the --parallel flag.

Steps to reproduce

Run the following program. Even though parallelism is unbounded by default, the maximum number of concurrently created resources is limited.

"""An AWS Python Pulumi program"""
import pulumi
import pulumi_aws as aws

# SQS
for i in range(100):
    name = f'pulumi-{str(i).rjust(3,"0")}'
    aws.sqs.Queue(
        name
    )
    
# SNS
for i in range(100):
    name = f'pulumi-{str(i).rjust(3,"0")}'
    aws.sns.Topic(
        name
    )

Expected Behavior

All resources should be created concurrently.

Actual Behavior

Only about 20 resources are created at a time.

Output of pulumi about

CLI          
Version      3.43.2-dev.0
Go Version   go1.19
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.18.0
python  unknown

Host     
OS       darwin
Version  12.6
Arch     arm64

This project is written in python: executable='/Users/robbiemckinstry/workspace/pulumi/pulumi/sdk/python/venv/bin/python3' version='3.10.6
'

Current Stack: dev

TYPE                  URN
pulumi:pulumi:Stack   urn:pulumi:dev::ls::pulumi:pulumi:Stack::ls-dev
pulumi:providers:aws  urn:pulumi:dev::ls::pulumi:providers:aws::default_5_18_0
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-001
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-000
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-003
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-002
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-004
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-006
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-005
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-007
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-009
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-008
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-010
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-011
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-012
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-013
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-014
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-015
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-016
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-017
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-018
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-019
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-020
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-021
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-022
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-023
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-024
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-025
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-026
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-027
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-029
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-028
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-030
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-031
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-032
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-033
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-034
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-035
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-036
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-037
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-038
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-039
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-040
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-041
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-042
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-043
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-044
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-045
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-046
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-047
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-048
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-049
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-050
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-051
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-052
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-053
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-054
aws:sqs/queue:Queue   urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-055


OPP TYPE  URN
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-056
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-057
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-058
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-059
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-060
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-061
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-062
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-063
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-064
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-065
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-066
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-067
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-068
creating  urn:pulumi:dev::ls::aws:sqs/queue:Queue::pulumi-069


Backend        
Name           pulumi.com
URL            https://app.pulumi.com/thesnowmancometh
User           thesnowmancometh
Organizations  thesnowmancometh, whiterabbit, pulumi

Dependencies:
NAME        VERSION
pip         22.3.0
pulumi-aws  5.18.0
setuptools  65.5.0
wheel       0.37.1

Pulumi locates its logs in /var/folders/8n/y4s73s2d4rnbn7d85clz9pz40000gn/T/ by default

Additional context

I'm currently working on a fix. This issue is also a program for the Automation API. I'm going to open a separate issue for that since I'm planning to separate the PRs for the respective fixes.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@RobbieMcKinstry RobbieMcKinstry added kind/bug Some behavior is incorrect or out of spec impact/performance Something is slower than expected language/python size/S Estimated effort to complete (1-2 days). area/language-host Runtime that executes user programs labels Oct 21, 2022
@RobbieMcKinstry RobbieMcKinstry added this to the 0.79 milestone Oct 21, 2022
@RobbieMcKinstry RobbieMcKinstry self-assigned this Oct 21, 2022
@RobbieMcKinstry RobbieMcKinstry linked a pull request Oct 22, 2022 that will close this issue
[SDK/Python] Respect --parallel flag #11122
Merged
3 tasks
@bors bors bot closed this as completed in #11122 Oct 22, 2022
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label Oct 22, 2022
@RobbieMcKinstry RobbieMcKinstry mentioned this issue Oct 24, 2022
Closed
@RobbieMcKinstry RobbieMcKinstry mentioned this issue Dec 8, 2022
Open
65 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RobbieMcKinstry RobbieMcKinstry

Labels
area/language-host Runtime that executes user programs impact/performance Something is slower than expected kind/bug Some behavior is incorrect or out of spec language/python resolution/fixed This issue was fixed size/S Estimated effort to complete (1-2 days).
Projects
None yet
Milestone
0.79
Development

Successfully merging a pull request may close this issue.

[SDK/Python] Respect --parallel flag pulumi/pulumi
2 participants
@RobbieMcKinstry @pulumi-bot