Performance difference between Python and Typescript

[ppawiggers]

I ran a benchmark deploying a Helm chart with Pulumi, using two languages: Python and Typescript. It seems that Typescript is quite a bit faster than the Python implementation. I just upped and destroyed a single Helm chart several times. The results:

• Typescript up: 199s / Typescript destroy: 478s
• Python up: 331s / Python destroy: 503s

So pulumi up with Typescript seems to be way faster than with Python. destroy takes even longer than up, however both need the same time. I also tried Go but that was evenly slow as Python.

In both cases the nodes had all images cached.

Where does this difference come from? Is it an anomaly in my tests, or is there a fundamental difference?

[lblackstone]

Where does this difference come from? Is it an anomaly in my tests, or is there a fundamental difference?

I can't say for sure without looking at tracing info, but my guess would be due to differences in the language runtimes, which can be exacerbated by stacks with more resources. In some cases, the pulumi up step also includes compiling the application before it runs, so there may be multiple factors at work here.

[karolzlot]

Results of creating and destroying bucket using gcp-native (direct rest communication) and templates from pulumi new:

google-native-python
google-native-typescript

Measured by stop-watch, so not very precise, but difference is visible anyway.

[RobbieMcKinstry]

Hello! I recently wrote a benchmark between Python and TypeScript. After merging #11122, I found Python was faster than TypeScript by a statistically significant margin. However, the absolute types are quite comparable, only differing by a few seconds (about 4%). In my benchmark, I tested at the scale of 200 resources created concurrently. I'd argue that Python and TypeScript perform similarly for nontrivial programs.

The performance difference described by @ppawiggers is for one resource, however, if for trivial programs. We capture nightly performance benchmarks for trivial programs to measure startup time and runtime for empty and trivial applies.

Here are some recent numbers:

	Python	TypeScript
S3 Bucket Empty	2953ms	2654ms
S3 Bucket Initial	7700ms	7099ms

IIRC, S3 Bucket Initial measures the amount of time it takes to spin up a single S3 bucket.
S3 Bucket Empty measures the time it takes to run up when the bucket has already been created. We recently audited these benchmarks for correctness, and they correspond fairly closely to the number we see on our local machines.

It seems like we've made a lot of strides in performance since Feb 20211. For that reason, I recommend closing this issue. Please feel free to comment if there are any objections and I'll reopen it 😸