Add `pulumi package` with subcommands `gen-sdk` and `get-schema` #10711

iwahbe · 2022-09-14T15:26:07Z

Description

Fixes #10693

This PR adds two commands that are vital to developing Pulumi packages without intermediary artifact and boilerplate.

gen-sdk: Takes either a package name, plugin path, or schema file and generates the associated SDK.
get-schema: Takes the same input as gen-sdk. It prints the schema out to stdout.

Checklist

I have added tests that prove my fix is effective or that my feature works

I have updated the CHANGELOG-PENDING file with my change

Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

pulumi-bot · 2022-09-14T15:31:03Z

Please view the results of the Downstream Codegen Tests Here

pulumi-bot · 2022-09-14T15:33:15Z

Please view the results of the Downstream Codegen Tests Here

pulumi-bot · 2022-09-14T15:37:33Z

Changelog

[uncommitted] (2022-09-15)

Features

[cli] Add a package author focused subcommand: pulumi package with subcommands pulumi package gen-sdk and pulumi package get-schema.
#10711

pulumi-bot · 2022-09-14T16:48:11Z

Please view the results of the Downstream Codegen Tests Here

pkg/cmd/pulumi/package.go

justinvp · 2022-09-15T00:28:53Z

pkg/cmd/pulumi/package_extract_schema.go

+
+func newExtractSchemaCommand() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "get-schema <schema_source>",


Wasn't there a proposal to offer CLI commands for invoking other plugin gRPC methods? If we're considering that, where would this command fit-in? cc: @Frassle

That proposal, AFAIK, was to directly call the provider commands, and print the output. This command (while very similar) does a little more than that. It verifies the schema is valid, round tripping it through our parsed representation, then formatting the output.

Agree with Ian, those commands are more intended for provider debugging, not for general user usage and while we would probably have a get-schema command under there as well for completeness it would just be a raw echo of the string sent back over grpc.

justinvp · 2022-09-15T00:30:10Z

pkg/cmd/pulumi/package_gen_sdk.go

+func genSDK(language, out string, pkg *schema.Package) error {
+	var f func(string, *schema.Package, map[string][]byte) (map[string][]byte, error)
+	switch language {
+	case "csharp", "c#", "dotnet":


Why are we accepting these additional names like "csharp" and "c#"?

We have a set of names for the folders we generate our SDKs into (nodejs, dotnet, python go, java) and a set of names we accept for pulumi convert (csharp, c#, go typescript, python, java). I'm accepting the union to avoid being confusing, but we will only emit our standard folders (dotnet correction, nodejs correction).

The main reason I asked is because we'll have to accommodate these various names when trying to select the right plugin when we move to a plugin model.

But if we're already doing it in pulumi convert, we'll have to be able to support these additional "aliases" there anyway, so no concern doing it here as well.

justinvp · 2022-09-15T00:43:29Z

pkg/cmd/pulumi/package_gen_sdk.go

+
+func genSDK(language, out string, pkg *schema.Package) error {
+	var f func(string, *schema.Package, map[string][]byte) (map[string][]byte, error)
+	switch language {


I wonder if this functionality should live in the language plugins. Then we wouldn't need to hard code this switch, we'd just load up the requested language plugin and call it over gRPC to do the SDK codegen, similar in spirit to #9294. The benefit being that if we add more languages in the future, we wouldn't necessarily have to touch this code again.

Not sure what that means for all, though, other than enumerating all available language plugins and invoking the appropriate gRPC method for each one.

Though, I can also understand possible pushback to this suggestion: that the codegen functionality doesn't really belong in the language plugins because their purpose (right now) is only about running Pulumi programs in the given language -- not about codegen. If others share this concern, we could consider another kind of plugin, maybe "codegen" (e.g. pulumi-codegen-nodejs, pulumi-codegen-python, etc.) that would be exclusively meant to handle codegen tasks (perhaps both sdkgen and programgen).

Of course, we could always hard code in a switch for now for expediency, and move to language plugin in the future.

Codegen can and should be pluginafied, but that is out of scope of this PR. Were hardcoding for now, but that is not the desired end goal.

I wonder if this functionality should live in the language plugins.

It should! We have discussed this multiple times, we will do this but it's probably tricky so keeps being put off.

justinvp · 2022-09-15T00:45:16Z

pkg/cmd/pulumi/package_gen_sdk.go

+	var language string
+	var out string
+	cmd := &cobra.Command{
+		Use:   "gen-sdk <schema_source>",


gen-sdk is probably the best verb I do wonder, though, if there are any other names we should consider like just generate or gen.

I considered generate, but it wasn't clear what was being generated (sdk, schema, binary, ...). I'm open to other suggestions though.

generate-sdk? I'd rather be clear here it's not like this is a command where terseness is really needed.

I'm good with gen-sdk. Looking over existing commands, we already have gen-markdown and gen-completion so there is precedent with just gen- (though, these are hidden).

pulumi-bot · 2022-09-15T09:31:33Z

Please view the results of the Downstream Codegen Tests Here

pulumi-bot · 2022-09-15T09:42:53Z

Please view the results of the Downstream Codegen Tests Here

iwahbe · 2022-09-15T09:44:54Z

I'm adding a no-changelog tag because the current CI system doesn't recognize the new changelog system.

justinvp

LGTM

iwahbe · 2022-09-16T17:54:10Z

bors r+

10711: Add `pulumi package` with subcommands `gen-sdk` and `get-schema` r=iwahbe a=iwahbe  # Description  Fixes #10693 This PR adds two commands that are vital to developing Pulumi packages without intermediary artifact and boilerplate. - `gen-sdk`: Takes either a package name, plugin path, or schema file and generates the associated SDK. - `get-schema`: Takes the same input as `gen-sdk`. It prints the schema out to stdout. ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works  - [x] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  10763: ci: Fix make brew r=AaronFriel a=AaronFriel This enables `make brew` to run as before, accounting for directory changes. The prep-for-goreleaser script places artifacts in `./bin/` now. `make brew` now succeeds. Fixes #10735 10767: [yaml] Update Pulumi YAML to 0.5.7 r=AaronFriel a=AaronFriel Co-authored-by: Ian Wahbe <ian@wahbe.com> Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>

bors · 2022-09-16T19:43:21Z

Build failed (retrying...):

bors-ok

10711: Add `pulumi package` with subcommands `gen-sdk` and `get-schema` r=iwahbe a=iwahbe  # Description  Fixes #10693 This PR adds two commands that are vital to developing Pulumi packages without intermediary artifact and boilerplate. - `gen-sdk`: Takes either a package name, plugin path, or schema file and generates the associated SDK. - `get-schema`: Takes the same input as `gen-sdk`. It prints the schema out to stdout. ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works  - [x] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  10772: Freeze `3.40.1` draft release. r=dixler a=dixler  # Description Bumping `.version` freezes the draft release. Fixes # (issue) ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works  - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  Co-authored-by: Ian Wahbe <ian@wahbe.com> Co-authored-by: Kyle Dixler <kyle@pulumi.com>

bors · 2022-09-16T23:53:06Z

Build failed (retrying...):

bors-ok

bors · 2022-09-17T03:15:16Z

Build succeeded:

bors-ok

iwahbe added 5 commits September 14, 2022 13:19

Begin on schema acquisition

510be1b

Get extract schema working

df2dbbd

Parse version

951d87c

Add gen-sdk sub-command

879f571

Provide diag sinks

7e8c6aa

iwahbe requested review from AaronFriel and justinvp September 14, 2022 15:26

iwahbe self-assigned this Sep 14, 2022

CL

6f98c5d

Fix lints

df558f4

justinvp reviewed Sep 15, 2022

View reviewed changes

Merge branch 'master' into iwahbe/10693/expose-sdkgen-from-cli

ec5db4d

iwahbe added 2 commits September 15, 2022 11:31

Update depreciated: ioutil -> os

6988a9d

CL (new style)

0f3198f

iwahbe added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Sep 15, 2022

iwahbe mentioned this pull request Sep 15, 2022

Expose SDK generation from pulumi #9721

Closed

3 tasks

iwahbe requested review from justinvp and Frassle September 15, 2022 12:59

justinvp approved these changes Sep 16, 2022

View reviewed changes

bors bot merged commit 23212c9 into master Sep 17, 2022

bors bot deleted the iwahbe/10693/expose-sdkgen-from-cli branch September 17, 2022 03:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `pulumi package` with subcommands `gen-sdk` and `get-schema` #10711

Add `pulumi package` with subcommands `gen-sdk` and `get-schema` #10711

iwahbe commented Sep 14, 2022 •

edited

pulumi-bot commented Sep 14, 2022

pulumi-bot commented Sep 14, 2022

pulumi-bot commented Sep 14, 2022 •

edited

pulumi-bot commented Sep 14, 2022

justinvp Sep 15, 2022

iwahbe Sep 15, 2022

Frassle Sep 16, 2022

justinvp Sep 15, 2022

iwahbe Sep 15, 2022

justinvp Sep 16, 2022

justinvp Sep 15, 2022

iwahbe Sep 15, 2022

Frassle Sep 16, 2022

justinvp Sep 15, 2022

iwahbe Sep 15, 2022

Frassle Sep 16, 2022

justinvp Sep 16, 2022

pulumi-bot commented Sep 15, 2022

pulumi-bot commented Sep 15, 2022

iwahbe commented Sep 15, 2022

justinvp left a comment

iwahbe commented Sep 16, 2022

bors bot commented Sep 16, 2022

bors bot commented Sep 16, 2022

bors bot commented Sep 17, 2022

Add pulumi package with subcommands gen-sdk and get-schema #10711

Add pulumi package with subcommands gen-sdk and get-schema #10711

Conversation

iwahbe commented Sep 14, 2022 • edited

Description

Checklist

pulumi-bot commented Sep 14, 2022

pulumi-bot commented Sep 14, 2022

pulumi-bot commented Sep 14, 2022 • edited

Changelog

[uncommitted] (2022-09-15)

Features

pulumi-bot commented Sep 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pulumi-bot commented Sep 15, 2022

pulumi-bot commented Sep 15, 2022

iwahbe commented Sep 15, 2022

justinvp left a comment

Choose a reason for hiding this comment

iwahbe commented Sep 16, 2022

bors bot commented Sep 16, 2022

bors bot commented Sep 16, 2022

bors bot commented Sep 17, 2022

Add `pulumi package` with subcommands `gen-sdk` and `get-schema` #10711

Add `pulumi package` with subcommands `gen-sdk` and `get-schema` #10711

iwahbe commented Sep 14, 2022 •

edited

pulumi-bot commented Sep 14, 2022 •

edited