Add support for remote_files for http_archive #22155

fzakaria · 2024-04-26T17:59:47Z

This issue adds support necessary to tackle
bazelbuild/bazel-central-registry#1566

Add two new attributes to http_archive: remote_file_urls and remote_file_integrity.

The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files.

This means we could probably deprecate build_file since that could be referenced as a file:// url in the remote_file_urls attribute.

Co-authored-by: Mark Williams markrwilliams@google.com

CC @fmeum

google-cla · 2024-04-26T17:59:52Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

tools/build_defs/repo/utils.bzl

src/test/shell/bazel/external_remote_file_test.sh

fzakaria · 2024-04-26T18:01:27Z

tools/build_defs/repo/http.bzl

@@ -139,6 +140,8 @@ def _http_archive_impl(ctx):
    auth = get_auth(ctx, all_urls)

    download_info = ctx.download_and_extract(
+        # TODO(fzakaria): all_urls here has the remote_patch URL which is incorrect


@fmeum I think this is a bug -- should I cut one?

Yeah, this seems wrong to me. This was likely an oversight in 445dfab, which just wanted to get this into get_auth.

CC @Wyverald

Filed #22201 for tracking.

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <markrwilliams@google.com>

* Added async download for remote_file for http-archive via block param * Added validation to check that there is no relative paths present

* Added disallow relative path * Added disallow absolute path * Added test for missing integrity

fmeum

LGTM with minor comments

tools/build_defs/repo/utils.bzl

src/test/shell/bazel/external_remote_file_test.sh

* Fixup comments * Add test that verifies remote_files works with more than 1 segment

src/test/shell/bazel/external_remote_file_test.sh

fmeum

@Wyverald Could you do the owner's review?

tools/build_defs/repo/http.bzl

Wyverald

very nice! just a few docs nits.

tools/build_defs/repo/http.bzl

tools/build_defs/repo/utils.bzl

fmeum · 2024-05-07T06:14:00Z

@bazel-io fork 7.2.0

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <markrwilliams@google.com> CC @fmeum Closes bazelbuild#22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <markrwilliams@google.com> CC @fmeum Closes #22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff Commit e01509f Co-authored-by: Farid Zakaria <fmzakari@google.com>

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <markrwilliams@google.com> CC @fmeum Closes bazelbuild#22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff

This is a continuation of bazelbuild#22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality. The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files. bazelbuild/bazel-central-registry#1566 has a good discussion of the rationale.

github-actions bot added the awaiting-review PR is awaiting review from an assigned reviewer label Apr 26, 2024

fzakaria commented Apr 26, 2024

View reviewed changes

fzakaria mentioned this pull request Apr 26, 2024

[Idea] Support Starlark overlays without patches bazelbuild/bazel-central-registry#1566

Open

fzakaria force-pushed the overlay-pr branch from 2899c6e to 840c498 Compare April 26, 2024 18:04

iancha1992 added the team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. label Apr 26, 2024

fzakaria mentioned this pull request Apr 30, 2024

http.bzl : all_urls is given the remote_patch URL which is surely a mistake. #22201

Open

fzakaria and others added 5 commits April 30, 2024 21:03

Update the MODULE.bazel.lock for test_repos

665f2ba

Remove spawn_strategy on account of bazelbuild#20753

7eab852

Added validation and async for remote_file

2e90536

* Added async download for remote_file for http-archive via block param * Added validation to check that there is no relative paths present

Updated lock files

a81db3c

fzakaria force-pushed the overlay-pr branch from d29d011 to a81db3c Compare April 30, 2024 21:09

fzakaria added 4 commits May 1, 2024 00:04

update MODULE.bazel.lock

4f150b9

Add back windows and macos python extension in MODULE.bazel.lock

8a0b673

Added more tests

d0d2970

* Added disallow relative path * Added disallow absolute path * Added test for missing integrity

Added test for bad integrity

1ebfbd1

fzakaria requested a review from fmeum May 1, 2024 18:06

fmeum approved these changes May 3, 2024

View reviewed changes

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Outdated Show resolved Hide resolved

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Outdated Show resolved Hide resolved

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Show resolved Hide resolved

fzakaria added 3 commits May 3, 2024 16:35

Address feedback from fmeum

9f61f11

* Fixup comments * Add test that verifies remote_files works with more than 1 segment

Merge branch 'master' into overlay-pr

9b897bd

Update lock files

cf42f89

fzakaria requested a review from fmeum May 3, 2024 16:40

Add shard count to build rule

89e4878

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Show resolved Hide resolved

fzakaria requested a review from fmeum May 3, 2024 18:25

fmeum approved these changes May 3, 2024

View reviewed changes

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

fzakaria added 3 commits May 6, 2024 17:30

Fixup docstring for remote_file_* attributes

6e558c0

Merge branch 'master' into overlay-pr

f330333

Update lock files

3cc59bb

fzakaria requested a review from fmeum May 6, 2024 17:40

update MODULE.bazel.lock

c91faf7

Wyverald approved these changes May 6, 2024

View reviewed changes

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/utils.bzl Outdated Show resolved Hide resolved

Address feedback in PR for comments

e3ad0a9

fzakaria requested a review from Wyverald May 6, 2024 21:34

Wyverald approved these changes May 6, 2024

View reviewed changes

Wyverald added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels May 6, 2024

bazel-io mentioned this pull request May 7, 2024

[7.2.0] Add support for remote_files for http_archive #22271

Closed

copybara-service bot closed this in e01509f May 10, 2024

github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label May 10, 2024

bazel-io mentioned this pull request May 10, 2024

[7.2.0] Add support for remote_files for http_archive #22325

Merged

fzakaria mentioned this pull request May 13, 2024

Add support for remote_files to bzlmod #22349

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for remote_files for http_archive #22155

Add support for remote_files for http_archive #22155

fzakaria commented Apr 26, 2024 •

edited

google-cla bot commented Apr 26, 2024

fzakaria Apr 26, 2024

fmeum Apr 30, 2024

fzakaria Apr 30, 2024

fmeum left a comment

fmeum left a comment

Wyverald left a comment

fmeum commented May 7, 2024

Add support for remote_files for http_archive #22155

Add support for remote_files for http_archive #22155

Conversation

fzakaria commented Apr 26, 2024 • edited

google-cla bot commented Apr 26, 2024

fzakaria Apr 26, 2024

Choose a reason for hiding this comment

fmeum Apr 30, 2024

Choose a reason for hiding this comment

fzakaria Apr 30, 2024

Choose a reason for hiding this comment

fmeum left a comment

Choose a reason for hiding this comment

fmeum left a comment

Choose a reason for hiding this comment

Wyverald left a comment

Choose a reason for hiding this comment

fmeum commented May 7, 2024

fzakaria commented Apr 26, 2024 •

edited