feat: go_repository supports netrc #848

qzmfranklin · 2020-07-28T16:01:41Z

What type of PR is this?
Feature

What package or component does this PR mostly affect?

go_repository

What does this PR do? Why is it needed?

This PR adds the netrc and auth_patterns attributes to the go_repository rule.

This allows go_repository to be used with downloading servers that require login credentials.

This also makes go_repository more closely resemble the interface of http_archive.

Which issues(s) does this PR fix?

Fixes #847

Other notes for review

This patch was largely the result of directly pirating from the source code of http_archive. It may not be the most beautiful code ever.

This patch has been tested on my local machine with a custom http server that requires login. I successfully re-used the same netrc file for go_repository and curl (with --netrc-file).

If more tests are needed, please let me know.

qzmfranklin · 2020-07-28T16:19:13Z

@jayconrod

I'm a bit surprised that this use case has not been covered yet.

I ran into this problem because I had a personal git server that requires http login-password credentials for downloading archives.

The netrc file I used is like this:

machine git.corp.internaldomain.com
login myusername
password mypassword

I used this netrc file with curl --netrc-file, the requests python library, the http_archive rule, and this patched go_repository rule. They all yield the same and expected results.

jayconrod · 2020-08-03T21:17:28Z

internal/go_repository.bzl

+load(
+    "@bazel_tools//tools/build_defs/repo:utils.bzl",
+    "read_netrc",
+    "use_netrc",


Could you confirm that Bazel 1.2.0 provides these functions, and if not, what is the oldest version of Bazel that does? I'd like to make sure we're not breaking compatibility here. If these were added recently, it would be better to copy them here.

jayconrod · 2020-08-03T21:20:55Z

internal/go_repository.bzl

+
+    if "HOME" in ctx.os.environ and not ctx.os.name.startswith("windows"):
+        netrcfile = "%s/.netrc" % (ctx.os.environ["HOME"])
+        if ctx.execute(["test", "-f", netrcfile]).return_code == 0:


Why use test here but ctx.path(netrcfile).exists below? Seems like the latter should work in both cases.

jayconrod · 2020-08-03T21:22:04Z

internal/go_repository.bzl

+def _get_auth(ctx, urls):
+    """Given the list of URLs obtain the correct auth dict."""
+    if ctx.attr.netrc:
+        netrc = read_netrc(ctx, ctx.attr.netrc)


To avoid repetition, please separate the code that decides which file to read from the calls to read_netrc and use_netrc.

jayconrod · 2020-08-03T21:23:25Z

internal/go_repository.bzl

+        netrc = read_netrc(ctx, ctx.attr.netrc)
+        return use_netrc(netrc, urls, ctx.attr.auth_patterns)
+
+    if "HOME" in ctx.os.environ and not ctx.os.name.startswith("windows"):


Before looking for a .netrc file in the home directory, check the NETRC environment variable.

jayconrod · 2020-08-03T21:23:48Z

internal/go_repository.bzl

+            return use_netrc(netrc, urls, ctx.attr.auth_patterns)
+
+    if "USERPROFILE" in ctx.os.environ and ctx.os.name.startswith("windows"):
+        netrcfile = "%s/.netrc" % (ctx.os.environ["USERPROFILE"])


On Windows, it should be _netrc, not .netrc.

jayconrod · 2020-08-03T21:24:20Z

internal/go_repository.bzl

@@ -214,6 +241,8 @@ go_repository = repository_rule(
        "strip_prefix": attr.string(),
        "type": attr.string(),
        "sha256": attr.string(),
+        "netrc": attr.string(),
+        "auth_patterns": attr.string_dict(),


Please add documentation to go_repository.bzl for these attributes.

jayconrod · 2020-08-03T21:26:47Z

internal/go_repository.bzl

 load("@io_bazel_rules_go//go/private:common.bzl", "env_execute", "executable_extension")
 load("@bazel_gazelle//internal:go_repository_cache.bzl", "read_cache_env")

 # We can't disable timeouts on Bazel, but we can set them to large values.
 _GO_REPOSITORY_TIMEOUT = 86400

+def _get_auth(ctx, urls):


I expect folks will want to have just one netrc file shared by all rules, so let's add these attributes to go_repository_config as well and make sure they can be read by go_repository if they're set there.

jayconrod · 2020-08-03T21:30:35Z

internal/go_repository.bzl

        ctx.download_and_extract(
            url = ctx.attr.urls,
            sha256 = ctx.attr.sha256,
            stripPrefix = ctx.attr.strip_prefix,
            type = ctx.attr.type,
+            auth = auth,


It looks like this will only work for HTTP mode. It also needs to work for module mode though (the ctx.attr.version branch).

I think that just means making sure NETRC is set to the correct file in env_keys, but please verify.

dmivankov · 2021-01-22T09:47:07Z

Does seem to work with Bazel 3.3.1 on github private repos like this

go_repository(
    name = "...",
    auth_patterns = {
        "api.github.com": "Bearer <password>",
    },
    importpath = "github.com/org/repo",
    strip_prefix = "org-repo-commit_here",
    type = "zip", # zipball -> tarball and zip -> tar should also work
    urls = ["https://api.github.com/repos/org/repo/zipball/commit_here"],
)

plus ~/.netrc

machine api.github.com
  password github_personal_token
  login anything

linzhp · 2021-08-10T15:13:18Z

Superseded by #1090

Initial PR to handle netrc had support for auth_patterns bazelbuild#848 But final version didn't bazelbuild#1090 auth_patterns can still be needed, for example for private github repositories Can be used like this ```python go_repository( name = "something", auth_patterns = { "api.github.com": "Bearer <password>", }, importpath = "github.com/someorg/something", strip_prefix = "something-...", type = "zip", urls = ["https://api.github.com/repos/someorg/something/zipball/..."], ) ``` Copying parameter documentation from bazel code

Initial PR to handle netrc had support for auth_patterns #848 But final version didn't #1090 auth_patterns can still be needed, for example for private github repositories Can be used like this ```python go_repository( name = "something", auth_patterns = { "api.github.com": "Bearer <password>", }, importpath = "github.com/someorg/something", strip_prefix = "something-...", type = "zip", urls = ["https://api.github.com/repos/someorg/something/zipball/..."], ) ```

feat: go_repository supports netrc

8ebec79

qzmfranklin requested a review from jayconrod as a code owner July 28, 2020 16:01

jayconrod reviewed Aug 3, 2020

View reviewed changes

blico mentioned this pull request Aug 5, 2021

Read .netrc in go_repository #1090

Merged

linzhp closed this Aug 10, 2021

dmivankov mentioned this pull request May 18, 2022

Add support for auth_patterns in go_repository #1254

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: go_repository supports netrc #848

feat: go_repository supports netrc #848

qzmfranklin commented Jul 28, 2020

qzmfranklin commented Jul 28, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

jayconrod Aug 3, 2020

dmivankov commented Jan 22, 2021

linzhp commented Aug 10, 2021

feat: go_repository supports netrc #848

feat: go_repository supports netrc #848

Conversation

qzmfranklin commented Jul 28, 2020

qzmfranklin commented Jul 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmivankov commented Jan 22, 2021

linzhp commented Aug 10, 2021