Yarn writes registry.yarnpkg.com to lockfile when private scoped packages is from registry.npmjs.org

[stieg]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I have something like the following in my ~/.npmrc file:

@mycompany:registry=https://registry.npmjs.org/
//registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

This is necessary so I can login to my private repo and get the packages I need to build locally.

When I add a new package like @mycompany/foo I would expect an entry in the yarn.lock file to look something like this:

"@mycompany/foo@^1.1.0":
  version "1.1.0"
  resolved "https://registry.npmjs.org/@mycompany/foo/-/foo-1.1.0.tgz#103034ba565c288eb781b1fd1065c9f05dfb90f9"

Yeah... not so much. Instead it looks like this:

"@mycompany/foo@^1.1.0":
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/@mycompany/foo/-/foo-1.1.0.tgz#103034ba565c288eb781b1fd1065c9f05dfb90f9"

Uncool yarn. This causes my CI to break because it will try and dig up the packages at registry.yarnpkg.com. Since this is a private registry the login will fail (since the registry is hosted on registry.npmjs.org) and thus it will cause other systems to fail to install packages.

Here is how to reproduce:

1. Setup a private repo on npm with some namespace
2. Set your .npmrc file to something like the one I provided above (except you replace the scope mycompany with your own scope, and set a valid auth token of course), and run yarn add <your scoped package>. You can do this by doing npm login --registry=https://registry.npmjs.org --scope=@<your scope name>. I would use yarn login but that doesn't work :(
3. Add a package from your scope using yarn add
4. Push a trivial update to that package in your scope.
5. Tell yarn to upgrade the package.

You will see that the URL gets changed in the lock file.

Please mention your node.js, yarn and operating system version.
node: 8.2.1
yarn: 0.28.4
OS: Fedora 26

[stieg]

Bump Anyone else seeing this behavior? No reply from anyone so far :(

[Haroenv]

Should be fixable by implementing #3330

[stieg]

@Haroenv Yep... should be fixed by #3330 but that is going to break backwards compatibility and based on discussion in that thread that won't be resolved for some time (need RFC, general agreement, roll out of a major release).

A simpler fix here AFAICT would involve disabling yarns predisposition to replace registry.npmjs.org with registry.yarnpkg.com. I'm not sure why Yarn is so very keen on doing this as it seems to have created plenty of issues (I should cite them here... but I don't want to spend the time to dig them up so will not do so but can if needed). By removing that silly behavior this problem here goes away.

[karlringrose]

+1 I am finding this issue as well. have you gotten any work around to effectively work for you @stieg ?

[stieg]

Honestly we have moved back to npm for now since it works. My work around was a script that would modify the lock file with the correct path for those packages.

On Sun, Sep 10, 2017, 18:16 Karl Ringrose ***@***.***> wrote: I am finding this issue as well. have you gotten any work around to effectively work for you @stieg <https://github.com/stieg> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIozzeWZwvPAnbbh5r5GRugYWc6voD3ks5shLQJgaJpZM4O1Ps2> .

-- - Andrew S.

[karlringrose]

I was about to do this, though my work around now is to just delete the yarn.lock file, and have hard versions set within pkg for build. We will most likely migrate to npm@5 if this issue is not resolved.

[vinicius73]

This bug has been causing a lot of inconvenience for a long time.
Unfortunately we will have to go back to NPM 😢

[BYK]

I'm sorry that you are having issues with this. I'm trying to see exactly what is wrong since Yarn's registry is just a CDN that needs to bypass everything so it should work with scoped packages properly. In fact, I can locally use scoped packages such as @yarnpkg/lockfile with my npm credentials.

Can anyone provide me some step-by-step instructions to replicate the issue locally and then we can start working on a solution? This may simply be a bug when you say your default registry should be NPM and Yarn incorrectly replaces it with its default registry.

[stieg]

@BYK I would start by re-reading the first comment on this thread to get an understanding of the behavioural problem. The maybe not so clear subtext is that since Yarn is re-writing the lock file paths to registry.yarnpkg.com instead of registry.npmjs.org, yarn is trying to fetch packages from a private repository without any authentication (or the authentication is not going through the CDN; which ever one it is). This causes the failure described in step 1 (or at least I believe it to be the case). I believe the root of the problem is that Yarn has decided they need to automatically rewrite the registry.npmjs.org to registry.yarnpkg.com. This is no problem with public packages that require no auth but is a major one with those that do require auth. I hope that makes things a bit clearer.

[BYK]

@stieg - I've read the initial comment and have understood the issue. I'm just trying to understand how it happens since it doesn't happen to scoped packages for instance which require authentication.

I'm also investigating why the rewriting happens and if it is something that should happen in the presence of an explicit registry override.

That's why I was curious about new and precise step-by-step instructions. Also we've made a bunch of improvements about detecting when to send aithentication headers and when not to on the latest master. Would you be willing to test one of the latest nightlies and see if they already solve the issue (not the rewrite but the actual, fetching the package issue). Nightlies can be found here: https://yarnpkg.com/en/docs/nightly

[stieg]

I see @BYK; you need a way to reproduce this (as you mentioned in your first comment). When I first encountered this issue I was running yarn v0.28.4 (as noted in the first comment). It seems that I can only hit this issue with that version of yarn and lower. On the current 1.0.1 release things are broken in a different way for me (described below). Alas here are the requested steps to reproduce:

With yarn v0.28.4

1. Setup a private repo on npmjs.org as some scope. I am going to use the scope "foo" for this example.
2. Upload some bunk package. I am going to call this package "bar". When uploaded I refer to it as @foo/bar
3. Create a new project locally. I'm going to call this "baz".
4. Login to npmjs using npm login --scope=@foo command. I do this because yarn login does not work for private packages.
5. Add package bar as a dependency of baz. So yarn add @foo/bar. This should add the dependency to the yarn.lock file.
6. Blow away your node_modules directory. So rm -rf node_modules.
7. Nuke the yarn cache. yarn cache clean.
8. Try running yarn install. You will see that it fails with a message like this:

$ yarn install
yarn install v0.28.4
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@foo/bar/-/bar-0.0.1.tgz: Request failed \"404 Not Found\"".

That is the failure.
9. Run the following command to fix the lockfile path:

sed -ie 's,registry.yarnpkg.com/@foo,registry.npmjs.org/@foo,' yarn.lock

10. Run yarn install again. Observe that it works this time.

With yarn v1.0.1
Using the same steps from above I get stuck at step 5 with the following message:

$ yarn add @foo/bar
yarn add v1.0.1
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@foo/bar/-/bar-0.0.1.tgz: Request failed \"404 Not Found\"".

I can work around the problem by doing the following:

yarn config set registry https://registry.npmjs.org

This will cause the package to install successfully. But then I will get stuck with a diffent problem when I hit step 10 (yarn doesn't seem to be authenticating using the data from .npmrc). So I'm dead in the water :(. This behavior is the same with both 1.0.1 and the latest nightly.

Perhaps I am just doing something so very wrong that yarn is confused and thus not working? Tips appreciated :).

[BYK]

@stieg - I think we are onto something here :) I don't think it is what you are doing. 1.0.1 had still bugs around which would cause this. 1.0.2 is around the corner and you can test it with the nightlies https://yarnpkg.com/en/docs/nightly

I'm quite confident that the error will go away but I would appreciate your confirmation if you have the time.

[stieg]

1.0.1 had still bugs around which would cause this.

Any links to issues you are referring to? I tried the nightly yesterday to no avail so if there is a particular one you wish for me to check out I will be happy to do so.

[BYK]

@stieg - these three PRs: #4350, #4347, #4367

That said 1.0.2 is out with these fixes, so why don't you give that a spin and if it still fails share the full error log with --verbose?

[simonkberg]

I'm still seeing this on 1.0.2, but got it to work again by removing @scope:registry=https://registry.npmjs.org/ from .npmrc (only keeping //registry.npmjs.org/:_authToken=...).
Downgrading to 0.27.5 also works as expected.

[BYK]

@simonkberg thanks a lot for sharing these details. What I hear is @scope:registry=https://registry.npmjs.org/ doesn't work as expected. Is that right?

[simonkberg]

@BYK yep, in 1.0.2 that line still causes me to get 404 errors for our private scoped packages. It automatically gets added to your .npmrc when you authenticate to npm using npm login --scope=@scope

[BYK]

@simonkberg I've tried to reproduce this issue with the @yarnpkg scope and wasn't able to reproduce any problems at all. Only thing that comes to my mind is having an outdated yarn.lock file that was generated before the @scope:registry was set so the resolved URL is engraved there.

My suggestion would be to either update the lock file manually or remove it and regenerate with the new setting.

[stieg]

I just tried to reproduce and can confirm that 1.0.2 seems to resolve this issue. Thanks @BYK

[simonkberg]

Weird, I could reproduce this reliably on multiple of our repos before but I'm not able to anymore. I'll try to update our base/ci images to yarn 1.1.0 and if we encounter anymore authentication problems I'll report back. Thanks @BYK

[Bnaya]

I have this issue with 1.1.0
removing the

@myscope:registry=https://registry.npmjs.org/

did fix the issue, no idea why.

With and without that setting, yarn was able to access the package metadata,

verbose 0.85 Performing "GET" request to "https://registry.npmjs.org/@brandtotal%2flambda-common".
verbose 1.746 Request "https://registry.npmjs.org/@brandtotal%2flambda-common" finished with status code 200.

But with it, it failed with downloading the tgz file.

BTW, it would be nice if --verbose will also print info about if credentials sent with the request

[KidkArolis]

Opened a PR - #5162 feedback (or even QA) welcome.

[KidkArolis]

@stieg and others. At this point in time yarn still writes yarnpkg.com as registry to the lock files. But the auth issue has been fixed. That is it should be possible to install private npm pkgs in CI by authenticating against npm.

If this is still an issue for you, please reopen.

[evgenyfadeev]

I see this issue with yarn 1.7.0
Setting the registry url manually in the yarn.lock to https://registry.npmjs.org/ allows me to get around this issue.

[tbinna]

This is still an issue with yarn 1.22.10.

If I do yarn add @my-scope/my-private-package --verbose it says:

...
[1/4] 🔍  Resolving packages...
verbose 0.636523304 Performing "GET" request to "https://registry.npmjs.org/@my-scope%2fmy-private-package".
verbose 2.662684879 Request "https://registry.npmjs.org/@my-scope%2fmy-private-package" finished with status code 200.
...

In yarn.lock I get:

"@my-scope/my-private-package@...":
  resolved "https://registry.yarnpkg.com/@my-scope/my-private-package/..."

Which in turn fails in CI during yarn install with:

...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@my-scope/my-private-package/-/my-private-package-1.0.1.tgz: Request failed \"404 Not Found\"".
...

My project level .npmrc looks as follows:

@my-scope:registry=https://registry.npmjs.org/
//registry.npmjs.org/:_authToken=${NPM_AUTH_TOKEN}
//registry.npmjs.org/:always-auth=true