Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move InstallDependencies to the language plugin #9294

Merged
merged 16 commits into from Apr 3, 2022
Merged

Move InstallDependencies to the language plugin #9294

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
06892f8
Move InstallDependencies to the language plugin
Frassle Mar 23, 2022
c7cd867
Just warn if pty fails to open
Frassle Mar 25, 2022
d98f8b7
Add tests and return real tty files
Frassle Mar 25, 2022
4de3e1a
Merge remote-tracking branch 'origin/master' into moveInstallDependen…
Frassle Mar 25, 2022
b25f999
Add to CHANGELOG
Frassle Mar 25, 2022
9e6e620
lint
Frassle Mar 25, 2022
45cc65e
format
Frassle Mar 25, 2022
00e51a3
Merge remote-tracking branch 'origin/master' into moveInstallDependen…
Frassle Mar 26, 2022
6fa09ff
Test strings
Frassle Mar 26, 2022
ec18569
Merge remote-tracking branch 'origin/master' into moveInstallDependen…
Frassle Apr 2, 2022
74e477b
Log pty opening for trace debugging
Frassle Apr 2, 2022
e54302c
s/Hack/Workaround
Frassle Apr 2, 2022
dfbdb8c
Use termios
Frassle Apr 3, 2022
bf29a3c
Tweak terminal test
Frassle Apr 3, 2022
75ad940
lint
Frassle Apr 3, 2022
86aa170
Fix windows build
Frassle Apr 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
132 changes: 24 additions & 108 deletions pkg/cmd/pulumi/new.go
View file
Open in desktop
Expand Up @@ -20,7 +20,6 @@ import (
"fmt"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"runtime"
"sort"
Expand All @@ -38,15 +37,12 @@ import (
"github.com/pulumi/pulumi/sdk/v3/go/common/apitype"
"github.com/pulumi/pulumi/sdk/v3/go/common/diag/colors"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/config"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/executable"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/goversion"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/logging"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
"github.com/pulumi/pulumi/sdk/v3/nodejs/npm"
"github.com/pulumi/pulumi/sdk/v3/python"
)

type promptForValueFunc func(yes bool, valueType string, defaultValue string, secret bool,
Expand Down Expand Up @@ -254,6 +250,14 @@ func runNew(args newArgs) error {
proj.Name = tokens.PackageName(args.name)
proj.Description = &args.description
proj.Template = nil
// Hack for python, most of our templates don't specify a venv but we want to use one
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Hack/Workaround/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this actually will break some users, like Nix users. There are situations where users explicitly don't want to use a venv. If Python now always uses venv even when not called for, we should highlight this as a change in CHANGELOG.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah cool I was going to ask if we could just assume "venv" but that's a good reason not to. At any rate this isn't a breaking change because we already always set venv in "pulumi new"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IN that case it's completely fine ofc.

if proj.Runtime.Name() == "python" {
// If the template does give virtualenv use it, else default to "venv"
if _, has := proj.Runtime.Options()["virtualenv"]; !has {
proj.Runtime.SetOption("virtualenv", "venv")
}
}

if err = workspace.SaveProject(proj); err != nil {
return fmt.Errorf("saving project: %w", err)
}
Expand Down Expand Up @@ -283,7 +287,13 @@ func runNew(args newArgs) error {

// Install dependencies.
if !args.generateOnly {
if err := installDependencies(proj, root); err != nil {
projinfo := &engine.Projinfo{Proj: proj, Root: root}
pwd, _, ctx, err := engine.ProjectInfoContext(projinfo, nil, nil, cmdutil.Diag(), cmdutil.Diag(), false, nil)
if err != nil {
return err
}

if err := installDependencies(ctx, &proj.Runtime, pwd); err != nil {
return err
}
}
Expand Down Expand Up @@ -593,113 +603,19 @@ func saveConfig(stack backend.Stack, c config.Map) error {
}

// installDependencies will install dependencies for the project, e.g. by running `npm install` for nodejs projects.
func installDependencies(proj *workspace.Project, root string) error {
// TODO[pulumi/pulumi#1334]: move to the language plugins so we don't have to hard code here.
if strings.EqualFold(proj.Runtime.Name(), "nodejs") {
if bin, err := nodeInstallDependencies(); err != nil {
return fmt.Errorf("%s install failed; rerun manually to try again, "+
"then run 'pulumi up' to perform an initial deployment"+": %w", bin, err)

}
} else if strings.EqualFold(proj.Runtime.Name(), "python") {
return pythonInstallDependencies(proj, root)
} else if strings.EqualFold(proj.Runtime.Name(), "dotnet") {
return dotnetInstallDependenciesAndBuild(proj, root)
} else if strings.EqualFold(proj.Runtime.Name(), "go") {
if err := goInstallDependencies(); err != nil {
return err
}
}

return nil
}

// nodeInstallDependencies will install dependencies for the project or Policy Pack by running `npm install` or
// `yarn install`.
func nodeInstallDependencies() (string, error) {
fmt.Println("Installing dependencies...")
fmt.Println()

bin, err := npm.Install("", false /*production*/, os.Stdout, os.Stderr)
if err != nil {
return bin, err
}

fmt.Println("Finished installing dependencies")
fmt.Println()

return bin, nil
}

// pythonInstallDependencies will create a new virtual environment and install dependencies.
func pythonInstallDependencies(proj *workspace.Project, root string) error {
const venvDir = "venv"
if err := python.InstallDependencies(root, venvDir, true /*showOutput*/); err != nil {
return err
}

// Save project with venv info.
proj.Runtime.SetOption("virtualenv", venvDir)
if err := workspace.SaveProject(proj); err != nil {
return fmt.Errorf("saving project: %w", err)
}
return nil
}

// dotnetInstallDependenciesAndBuild will install dependencies and build the project.
func dotnetInstallDependenciesAndBuild(proj *workspace.Project, root string) error {
contract.Assert(proj != nil)

fmt.Println("Installing dependencies...")
fmt.Println()

projinfo := &engine.Projinfo{Proj: proj, Root: root}
pwd, main, plugctx, err := engine.ProjectInfoContext(projinfo, nil, nil, cmdutil.Diag(), cmdutil.Diag(), false, nil)
func installDependencies(ctx *plugin.Context, runtime *workspace.ProjectRuntimeInfo, directory string) error {
// First make sure the language plugin is present. We need this to load the required resource plugins.
// TODO: we need to think about how best to version this. For now, it always picks the latest.
lang, err := ctx.Host.LanguageRuntime(runtime.Name())
if err != nil {
return err
return fmt.Errorf("failed to load language plugin %s: %w", runtime.Name(), err)
}
defer plugctx.Close()

// Call RunInstallPlugins, which will run `dotnet build`. This will automatically
// restore dependencies, install any plugins, and build the project so it will be
// prepped and ready to go for a faster initial `pulumi up`.
if err = engine.RunInstallPlugins(proj, pwd, main, nil, plugctx); err != nil {
return err
if err = lang.InstallDependencies(directory); err != nil {
return fmt.Errorf("installing dependencies failed; rerun manually to try again, "+
"then run 'pulumi up' to perform an initial deployment: %w", err)
}

fmt.Println("Finished installing dependencies")
fmt.Println()

return nil
}

// goInstallDependencies will install dependencies for the project by running `go mod tidy`.
func goInstallDependencies() error {
fmt.Println("Installing dependencies...")
fmt.Println()

gobin, err := executable.FindExecutable("go")
if err != nil {
return err
}

if err = goversion.CheckMinimumGoVersion(gobin); err != nil {
return err
}

cmd := exec.Command(gobin, "mod", "tidy")
cmd.Env = os.Environ()
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr

if err := cmd.Run(); err != nil {
return fmt.Errorf("`go mod tidy` failed to install dependencies; rerun manually to try again, "+
"then run 'pulumi up' to perform an initial deployment"+": %w", err)

}

fmt.Println("Finished installing dependencies")
fmt.Println()

return nil
}

Expand Down
10 changes: 9 additions & 1 deletion pkg/cmd/pulumi/policy_new.go
View file
Open in desktop
Expand Up @@ -26,6 +26,7 @@ import (
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
"github.com/pulumi/pulumi/sdk/v3/nodejs/npm"
"github.com/pulumi/pulumi/sdk/v3/python"
"github.com/spf13/cobra"
survey "gopkg.in/AlecAivazis/survey.v1"
Expand Down Expand Up @@ -189,9 +190,16 @@ func runNewPolicyPack(args newPolicyArgs) error {
func installPolicyPackDependencies(proj *workspace.PolicyPackProject, projPath, root string) error {
// TODO[pulumi/pulumi#1334]: move to the language plugins so we don't have to hard code here.
if strings.EqualFold(proj.Runtime.Name(), "nodejs") {
if bin, err := nodeInstallDependencies(); err != nil {
fmt.Println("Installing dependencies...")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we need to use logging instead of fmt.Println?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is copied from nodeInstallDependencies which already used fmt.Println

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's moving processes though right. The plugin executable runs in a process where stdout is used to communicate the port on which the engine is running. Printing after that might work. I'm just careful. Hopefully the tests cover us here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope this hasn't moved process

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah never mind. Perfect ty. pkg/cmd you're right.

fmt.Println()

bin, err := npm.Install("", false /*production*/, os.Stdout, os.Stderr)
if err != nil {
return fmt.Errorf("`%s install` failed; rerun manually to try again.: %w", bin, err)
}

fmt.Println("Finished installing dependencies")
fmt.Println()
} else if strings.EqualFold(proj.Runtime.Name(), "python") {
const venvDir = "venv"
if err := python.InstallDependencies(root, venvDir, true /*showOutput*/); err != nil {
Expand Down
9 changes: 8 additions & 1 deletion pkg/cmd/pulumi/up.go
View file
Open in desktop
Expand Up @@ -307,7 +307,14 @@ func newUpCmd() *cobra.Command {
}

// Install dependencies.
if err = installDependencies(proj, root); err != nil {

projinfo := &engine.Projinfo{Proj: proj, Root: root}
pwd, _, ctx, err := engine.ProjectInfoContext(projinfo, nil, nil, cmdutil.Diag(), cmdutil.Diag(), false, nil)
if err != nil {
return result.FromError(fmt.Errorf("building project context: %w", err))
}

if err = installDependencies(ctx, &proj.Runtime, pwd); err != nil {
return result.FromError(err)
}

Expand Down
1 change: 1 addition & 0 deletions pkg/go.mod
View file
Open in desktop
Expand Up @@ -163,6 +163,7 @@ require (
)

require (
github.com/creack/pty v1.1.9 // indirect
github.com/hashicorp/go-version v1.4.0 // indirect
github.com/rogpeppe/go-internal v1.8.1 // indirect
)
4 changes: 4 additions & 0 deletions pkg/resource/deploy/deploytest/languageruntime.go
View file
Open in desktop
Expand Up @@ -65,3 +65,7 @@ func (p *languageRuntime) Run(info plugin.RunInfo) (string, bool, error) {
func (p *languageRuntime) GetPluginInfo() (workspace.PluginInfo, error) {
return workspace.PluginInfo{Name: "TestLanguage"}, nil
}

func (p *languageRuntime) InstallDependencies(directory string) error {
return nil
}
36 changes: 36 additions & 0 deletions sdk/dotnet/cmd/pulumi-language-dotnet/main.go
View file
Open in desktop
Expand Up @@ -33,6 +33,7 @@ import (
"github.com/pkg/errors"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/executable"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/logging"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/rpcutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/version"
Expand Down Expand Up @@ -660,3 +661,38 @@ func (host *dotnetLanguageHost) GetPluginInfo(ctx context.Context, req *pbempty.
Version: version.Version,
}, nil
}

func (host *dotnetLanguageHost) InstallDependencies(
req *pulumirpc.InstallDependenciesRequest, server pulumirpc.LanguageRuntime_InstallDependenciesServer) error {

stdout, stderr, err := rpcutil.MakeStreams(server, req.IsTerminal)
if err != nil {
return err
}
// best effort close, but we try an explicit close and error check at the end as well
defer stdout.Close()

stdout.Write([]byte("Installing dependencies...\n\n"))

dotnetbin, err := executable.FindExecutable("dotnet")
if err != nil {
return err
}

cmd := exec.Command(dotnetbin, "build")
cmd.Dir = req.Directory
cmd.Env = os.Environ()
cmd.Stdout, cmd.Stderr = stdout, stderr

if err := cmd.Run(); err != nil {
return fmt.Errorf("`dotnet build` failed to install dependencies: %w", err)

}
stdout.Write([]byte("Finished installing dependencies\n\n"))

if err := stdout.Close(); err != nil {
return err
}

return nil
}
1 change: 1 addition & 0 deletions sdk/go.mod
View file
Open in desktop
Expand Up @@ -8,6 +8,7 @@ require (
github.com/Microsoft/go-winio v0.4.14
github.com/blang/semver v3.5.1+incompatible
github.com/cheggaaa/pb v1.0.18
github.com/creack/pty v1.1.9
github.com/djherbis/times v1.2.0
github.com/gofrs/uuid v3.3.0+incompatible
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
Expand Down
1 change: 1 addition & 0 deletions sdk/go.sum
View file
Open in desktop
Expand Up @@ -31,6 +31,7 @@ github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
github.com/creack/pty v1.1.9 h1:uDmaGzcdjhF4i/plgjmEsriH11Y0o7RKapEf/LDaM3w=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
Expand Down
3 changes: 3 additions & 0 deletions sdk/go/common/resource/plugin/langruntime.go
View file
Open in desktop
Expand Up @@ -38,6 +38,9 @@ type LanguageRuntime interface {
Run(info RunInfo) (string, bool, error)
// GetPluginInfo returns this plugin's information.
GetPluginInfo() (workspace.PluginInfo, error)

// InstallDependencies will install dependencies for the project, e.g. by running `npm install` for nodejs projects.
InstallDependencies(directory string) error
}

// ProgInfo contains minimal information about the program to be run.
Expand Down
52 changes: 52 additions & 0 deletions sdk/go/common/resource/plugin/langruntime_plugin.go
View file
Open in desktop
Expand Up @@ -16,6 +16,8 @@ package plugin

import (
"fmt"
"io"
"os"
"path/filepath"
"strings"

Expand All @@ -24,6 +26,7 @@ import (
"github.com/pkg/errors"
"google.golang.org/grpc/codes"

"github.com/pulumi/pulumi/sdk/v3/go/common/diag/colors"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
Expand Down Expand Up @@ -236,3 +239,52 @@ func (h *langhost) Close() error {
}
return nil
}

func (h *langhost) InstallDependencies(directory string) error {
logging.V(7).Infof("langhost[%v].InstallDependencies(directory=%s) executing",
h.runtime, directory)
resp, err := h.client.InstallDependencies(h.ctx.Request(), &pulumirpc.InstallDependenciesRequest{
Directory: directory,
IsTerminal: cmdutil.GetGlobalColorization() != colors.Never,
})

if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].InstallDependencies(directory=%s) failed: err=%v",
h.runtime, directory, rpcError)

// It's possible this is just an older language host, prior to the emergence of the InstallDependencies
// method. In such cases, we will silently error (with the above log left behind).
if rpcError.Code() == codes.Unimplemented {
return nil
}

return rpcError
}

for {
output, err := resp.Recv()
if err != nil {
if err == io.EOF {
break
}
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].InstallDependencies(directory=%s) failed: err=%v",
h.runtime, directory, rpcError)
return rpcError
}

if len(output.Stdout) != 0 {
os.Stdout.Write(output.Stdout)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just sanity checking, langhost writes to stdout/stderr, or needs to "write" via gRPC?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh we use to call a subprocess and hook it's stdout to our stdout. Now we stream back stdout via grpc but we still want it to just write to stdout here.

}

if len(output.Stderr) != 0 {
os.Stderr.Write(output.Stderr)
}
}

logging.V(7).Infof("langhost[%v].InstallDependencies(directory=%s) success",
h.runtime, directory)
return nil

}