Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add process wrapper #257

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

jelmansouri
Copy link

As we are working toward having native rule compatibility with windows, we encountered a lot of run_shell usage to have access to some basic shell functionality like substituting $pwd in command line arguments or environment variables or redirecting the output. Or reading a file that contains arguments generated by another rule (build.rs scripts in rust).

@jelmansouri jelmansouri changed the title Process wrapper Add process wrapper Jul 8, 2020
@aiuto
Copy link
Contributor

aiuto commented Jul 8, 2020

Could you start with a design discussion somewhere?

I think I agree something like this is worthwhile because bash scripts have no place in a portable world, but there are some problems

  • this takes a dependency on C++, which I'm not sure we want in skylib
  • it's not clear the features implemented work in a general enough way to be useful outside of your use case.

@jelmansouri
Copy link
Author

I created a google doc to discuss this!
https://docs.google.com/document/d/13M9ArI_Qdj3PuJk2kXSRm6YISp8uceq_lEG8_I7kXcA

@jin
Copy link
Member

jin commented Jul 9, 2020

cc @meteorcloudy

@aiuto
Copy link
Contributor

aiuto commented Jul 9, 2020 via email

docs/process_wrapper_doc.md Show resolved Hide resolved
docs/process_wrapper_doc.md Show resolved Hide resolved
docs/process_wrapper_doc.md Show resolved Hide resolved
lib/process_wrapper/process_wrapper.cc Outdated Show resolved Hide resolved
docs/process_wrapper_doc.md Outdated Show resolved Hide resolved
docs/process_wrapper_doc.md Show resolved Hide resolved
docs/process_wrapper_doc.md Outdated Show resolved Hide resolved
@jmmv
Copy link
Contributor

jmmv commented Jul 9, 2020

Given that Bazel already has a process-wrapper tool, should this be called something else?

@jelmansouri
Copy link
Author

Given that Bazel already has a process-wrapper tool, should this be called something else?

I don't have any strong opinion about the name :) if you have a suggestion I would be happy to change it!

@jelmansouri
Copy link
Author

Updated the design doc with more information!

Copy link
Member

@meteorcloudy meteorcloudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll grant approval from Windows perspective.

Copy link
Contributor

@jmmv jmmv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Back to my comment about the name... but taking a step further: given that Bazel's process-wrapper already does a lot of what this does... maybe that one could be extended and reused in some way instead? Seems weird to have two very similar tools. (https://jmmv.dev/2019/11/bazel-process-wrapper.html)

Also, I'm wondering... all this argument rewriting... why cannot it be done from Starlark? I think the design doc should better explain why a separate tool is necessary. A separate tool feels a bit awkward to me, and it risks becoming its own "shell interpreter" over time.

docs/process_wrapper_doc.md Show resolved Hide resolved
lib/BUILD Outdated Show resolved Hide resolved
lib/process_wrapper/process_wrapper.cc Outdated Show resolved Hide resolved
return -1;
}

pid_t child_pid = fork();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to fork at all? From what I see in the description of this wrapper, there is no need to do anything upon process termination, so the wrapper can set things up and then give control to the wrapper process, keeping the process tree cleaner (and being much faster e.g. on macOS).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The touch file feature, needs getting back the control.

@jelmansouri
Copy link
Author

Also, I'm wondering... all this argument rewriting... why cannot it be done from Starlark? I think the design doc should better explain why a separate tool is necessary. A separate tool feels a bit awkward to me, and it risks becoming its own "shell interpreter" over time.

Ideally it would, but looking at some open issues dating back to 2018 like this one bazelbuild/bazel#5511, it might be harder to make these changes to support remote executions platfoms. But I'm not an expert here!

@aiuto aiuto removed their request for review July 10, 2020 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants