Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix execution requirements for 'build without the bytes' #639

Merged
merged 1 commit into from
Oct 31, 2023

Conversation

gregmagolan
Copy link
Member

@gregmagolan gregmagolan commented Oct 30, 2023

With "build without the bytes" working correctly and default in Bazel 7, our copy execution requirements should allow caching so that it works with this mode out of the box.

We should also not be opinionated about remote execution since with "build without the bytes", if you're copying all generated files around then it may be more efficient to do it on a remote executor. The subset of users that use remote execution can easily set custom execution requirements for the Mnenomic if desired.


Type of change

  • Performance (a code change that improves performance)

Test plan

  • Covered by existing test cases

@gregmagolan gregmagolan force-pushed the fixup_copy_exec_reqs branch 2 times, most recently from d5deb7a to 805ef9d Compare October 30, 2023 05:10
@gregmagolan gregmagolan changed the title fix: fix copy execution requirements for 'build without the bytes' fix: fix execution requirements for 'build without the bytes' Oct 30, 2023
@gregmagolan gregmagolan force-pushed the fixup_copy_exec_reqs branch 2 times, most recently from 5350ebb to 2e4c788 Compare October 30, 2023 09:36
Copy link
Member

@alexeagle alexeagle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to discuss this one, as we've gone back and forth several times now.

#594 points to #528 where we learned that some users rely on this.

We now have two modes in bazel-lib 2.0 https://github.com/aspect-build/bazel-lib/blob/main/docs/copy_file.md#choosing-execution-requirements

I don't understand whether your change here is a complete retreat from local performance opt.

@gregmagolan
Copy link
Member Author

gregmagolan commented Oct 30, 2023

It's not a retreat. It keeps the main local development optimization of no-sandbox which is the important one but it adds the ability remote cache the actions so that "build without the bytes" is optimal. It shaves off minutes of copying off of a new clone that is fully cached in a large rules_js repository with "build without the bytes" enabled.

@gregmagolan gregmagolan force-pushed the fixup_copy_exec_reqs branch 10 times, most recently from 2d9b19f to f8913cd Compare October 30, 2023 18:57
@alexeagle
Copy link
Member

Sure but do we understand how this new version satisfies all the problems we had with other attempts at this?

@gregmagolan
Copy link
Member Author

gregmagolan commented Oct 31, 2023

I understand how the requirements of "build without the bytes" fit in now after a deep dive last week. I think we should support that mode by default. Not allowing cacheing & remote execution is the wrong default IMO.

In terms of problems, performance with sandboxing was the big one which is the one default that makes sense to keep. If there was a way to not bother writing to the disk cache but still enable cacheing for the remote cache that would be nice but there is not AFAIK.

@gregmagolan gregmagolan merged commit a47eebf into main Oct 31, 2023
53 checks passed
@gregmagolan gregmagolan deleted the fixup_copy_exec_reqs branch October 31, 2023 20:38
@kzadorozhny
Copy link
Contributor

This is a very welcome change. Being able to speed up CopyFile actions is a big win.
For example, the total time spent copying 4K+ ts_priject source files is more than 220 seconds.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

4 participants