Abstract output manipulation (Apply/Lift/All) from Pulumi users

[toriancrane]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

The concept of Inputs and Outputs in Pulumi is not the most user friendly or intuitive. It can be very unclear when to use Apply (especially when Output.All is involved) vs Lifting and why one has to be used vs the other.

Ideally, a feature would be implemented to abstract these elements away so that users can interact with these Outputs/response objects as they normally would in their selected programming language.

For example, let's say I created a VPC using Python and I have the following response object:

{
    "assign_generated_ipv6_cidr_block": null,
    "availability_zone_names": null,
    "cidr_block": null,
    ...
    ...
    "private_subnet_ids": [
        "subnet-049cbd8d4a5c850d3",
        "subnet-0bc54f9331b10a986",
        "subnet-01e466d4adcb76db2"
    ],
    "public_subnet_ids": [
        "subnet-014468a24698efcc3",
        "subnet-0384f873694e9b2a7",
        "subnet-0e09b477488dbaad4"
    ],
    ...
    ...
    "vpc_id": "vpc-08e8aaf610070fa2e"
}

If I want to get the first subnet from the list of public subnet IDs, I should be able to retrieve it using the following:

vpc.public_subnet_ids[0]

Instead, I have to use the following syntax:

vpc.public_subnet_ids.apply(lambda id: id[0]))

No matter how nested a parameter is, what structure they take (String, List, Dict etc), or what type of manipulation is to be performed (return the value of a string versus build a string from multiple parameter values), a user should be able to interact with/manipulate it as they normally would in their selected language.

We currently have abstraction with Interpolation (Output.concat, Output.format, pulumi.interpolate etc). How can we do this at the level where apply and the inline lambda function inside of it is no longer needed?

Affected area/feature

The Pulumi service

[lukehoban]

If I want to get the first subnet from the list of public subnet IDs, I should be able to retrieve it using the following:

vpc.public_subnet_ids[0]

This does work today in Python and JavaScript, but in other languages requires some other sugar. One of the challenges/problems with these "lifting" approaches is that they are more inconsistent across languages (and even in some cases within a given language), which makes them harder to describe for a general Pulumi audience. But indeed this should work as you descirbe.

No matter how nested a parameter is, what structure they take (String, List, Dict etc), or what type of manipulation is to be performed (return the value of a string versus build a string from multiple parameter values), a user should be able to interact with/manipulate it as they normally would in their selected language.

This is indeed the reason we support "lifting" to the degree possible in each language. However, in some languages it is not possible to "fake out" the underlying type system in this way.

We currently have abstraction with Interpolation (Output.concat, Output.format, pulumi.interpolate etc). How can we do this at the level where apply and the inline lambda function inside of it is no longer needed?

You can't within existing programming languages, because most languages don't offer the ability to at a deep enough level "pretend" something that is not a string is actually a string. When you create your own language you have the chance to define all function calls and operators as being fundamentally lifted over Output<T> - and to an extent that's what our Pulumi YAML "language" does - and why it doesn't need to worry about any of these things.

Ultimately, I'm not entirely sure whether there is any concrete suggestion we can pursue here yet. We have a few experiments that have largely led to dead-ends in things like https://github.com/pulumi/nooutput. We've also done some work in fleshing out a consistent set of "Output helpers" across languages to provide a consstent set of lifted functions for every language we can document and codegen so that these become standard in #11939. My best assumption for what this issue will ultimately track is that we follow through on completing that work, and then revise the Outputs documentation to embrace a consistent story about using lifted functions for each common use case. If there are other concrete things we should track, woudl be great to open issues on those.

[Frassle]

This is indeed the reason we support "lifting" to the degree possible in each language. However, in some languages it is not possible to "fake out" the underlying type system in this way.

Note that even in languages that do support that (like python and typescript) it makes an absolute mess of the typing annotations.