Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using corepack behind corporate proxy and injected certificate #417

Closed
adurand-ctie opened this issue Mar 6, 2024 · 16 comments · Fixed by #451
Closed

Using corepack behind corporate proxy and injected certificate #417

adurand-ctie opened this issue Mar 6, 2024 · 16 comments · Fixed by #451

Comments

@adurand-ctie
Copy link

Hi,
I'm willing to use API Platform (https://api-platform.com/) and the Dockerfile of the pwa component requires corepack:

RUN corepack enable && \
	corepack prepare --activate pnpm@latest && \
	pnpm config -g set store-dir /.pnpm-store

The base image is node:20-alpine.

I'm behind a corporate proxy, so I've added the following before using corepack:

RUN set HTTP_PROXY=http://proxy.etat.lu:8080
RUN set HTTPS_PROXY=http://proxy.etat.lu:8080
RUN set ALL_PROXY=http://proxy.etat.lu:8080

We also have a "traffic interceptor" that watches the whole traffic, so I added its certificate (which is already injected in the image):
RUN set NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/etat.crt

However, I'm still running into issues and receive the error below upon executing corepack prepare --activate pnpm@latest:

Internal Error: Error when performing the request to https://registry.npmjs.org/pnpm; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:42195:14)
    at ClientRequest.emit (node:events:518:28)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:518:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
The command '/bin/sh -c corepack prepare --activate pnpm@latest' returned a non-zero code: 1
ERROR: Service 'pwa' failed to build : Build failed

Anybody could help me figure out what I'm missing here?

Thanks in advance.
Adrien.

@merceyz
Copy link
Member

merceyz commented Mar 6, 2024

Probably related to nodejs/undici#2200.

@adurand-ctie
Copy link
Author

Probably related to nodejs/undici#2200.

To overcome this, should I try RUN set NODE_TLS_REJECT_UNAUTHORIZED=0?
If this is the case, it didn't work :(

@aduh95
Copy link
Contributor

aduh95 commented Mar 6, 2024

Could there be that you need to specify a COREPACK_NPM_REGISTRY in your environment?

@adurand-ctie
Copy link
Author

Still no luck, even with COREPACK_NPM_REGISTRY.
However, there is something weird. Because when I set this env variable to a fake value, like RUN set COREPACK_NPM_REGISTRY=https://www.microsoft.com/ (even between quotes), I still see that it does the request towards registry.npmjs.org:
Internal Error: Error when performing the request to https://registry.npmjs.org/pnpm; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting

@jlaminate
Copy link

Seeing the same issue here with corepack 0.25.x
NODE_EXTRA_CA_CERTS and HTTP_PROXY env variables work as expected in 0.24.x

@aureq
Copy link

aureq commented Mar 29, 2024

I've been digging a little more into this issue as I also use a proxy (squid-cache, no proxy auth, no tls inspection).

First, I don't believe this is an injected certificate issue since my proxy only relays connections out without any TLS inspection.

Second, since I fully control my local DNS server and the proxy, I have noticed no connection attempt is made to connect to the defined proxy. However, inspecting the agent object (https://github.com/nodejs/corepack/blob/v0.25.2/sources/httpUtils.ts#L11), it appears to be valid. So the error is fetch() is used (https://github.com/nodejs/corepack/blob/v0.25.2/sources/httpUtils.ts#L15)

Rolling backward, I tried to find what were the changes introduced between 0.24.1 (last known working version for me) and 0.25.0 (first version with that issue). Doing a git blame and I found the PR #365 introduced significant changes in how external data is fetched. Previously https.get(...)was used, but now it's globalThis.fetch(...). Either somehow globalThis.fetch() is broken, or the proxy agent isn't set as expected which causes the internal error.

Last, I found that downgrading to corepack 0.24.1 (npm install -g corepack@0.24.1), then running yarn --version will populate ~/.cache/node/corepack. I believe during this process, tags are retrieved and checked, and the latest known version is stored in ~/.cache/node/corepack/lastKnownGood.json

Then upgrading back to 0.25.2 and running the same command prompts the user to download and install yarn again. Answering yes leads to the same internal error as I described above. So it seems like using globalThis.fetch() breaks more corepack functionalities.

Tagging both @merceyz and @aduh95 who have authored the PR mentioned above for visibility.

When dumped, my ProxyAgent looked like this.

ProxyAgent {
  _events: [Object: null prototype] {},
  _eventsCount: 0,
  _maxListeners: undefined,
  [Symbol(shapeMode)]: false,
  [Symbol(kCapture)]: false,
  [Symbol(destroyed)]: false,
  [Symbol(onDestroyed)]: null,
  [Symbol(closed)]: false,
  [Symbol(onClosed)]: [],
  [Symbol(proxy agent options)]: { uri: 'http://proxy.local-network.net:3129', protocol: 'https' },
  [Symbol(proxy agent)]: Agent {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    [Symbol(shapeMode)]: false,
    [Symbol(kCapture)]: false,
    [Symbol(destroyed)]: false,
    [Symbol(onDestroyed)]: null,
    [Symbol(closed)]: false,
    [Symbol(onClosed)]: [],
    [Symbol(dispatch interceptors)]: [ [Function (anonymous)] ],
    [Symbol(options)]: {
      uri: 'http://proxy.local-network.net:3129',
      connect: [AsyncFunction: connect],
      interceptors: undefined
    },
    [Symbol(maxRedirections)]: 0,
    [Symbol(factory)]: [Function: defaultFactory],
    [Symbol(clients)]: Map(0) {},
    [Symbol(onDrain)]: [Function (anonymous)],
    [Symbol(onConnect)]: [Function (anonymous)],
    [Symbol(onDisconnect)]: [Function (anonymous)],
    [Symbol(onConnectionError)]: [Function (anonymous)]
  },
  [Symbol(dispatch interceptors)]: [],
  [Symbol(request tls settings)]: undefined,
  [Symbol(proxy tls settings)]: undefined,
  [Symbol(proxy headers)]: {},
  [Symbol(connect endpoint function)]: [Function: connect],
  [Symbol(proxy client)]: <ref *1> Pool {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    [Symbol(shapeMode)]: false,
    [Symbol(kCapture)]: false,
    [Symbol(destroyed)]: false,
    [Symbol(onDestroyed)]: null,
    [Symbol(closed)]: false,
    [Symbol(onClosed)]: [],
    [Symbol(queue)]: FixedQueue {
      tail: [FixedCircularBuffer],
      head: [FixedCircularBuffer]
    },
    [Symbol(clients)]: [],
    [Symbol(queued)]: 0,
    [Symbol(onDrain)]: [Function: onDrain],
    [Symbol(onConnect)]: [Function (anonymous)],
    [Symbol(onDisconnect)]: [Function (anonymous)],
    [Symbol(onConnectionError)]: [Function (anonymous)],
    [Symbol(stats)]: PoolStats { [Symbol(pool)]: [Circular *1] },
    [Symbol(dispatch interceptors)]: [],
    [Symbol(connections)]: null,
    [Symbol(url)]: URL {
      href: 'http://proxy.local-network.net:3129/',
      origin: 'http://proxy.local-network.net:3129',
      protocol: 'http:',
      username: '',
      password: '',
      host: 'proxy.local-network.net:3129',
      hostname: 'proxy.local-network.net',
      port: '3129',
      pathname: '/',
      search: '',
      searchParams: URLSearchParams {},
      hash: ''
    },
    [Symbol(options)]: {
      connect: [Function: connect],
      allowH2: undefined,
      interceptors: undefined
    },
    [Symbol(factory)]: [Function: defaultFactory]
  }
}

Hoping this will be helpful.

@marwin1991
Copy link

Seeing the same issue here with corepack 0.25.x NODE_EXTRA_CA_CERTS and HTTP_PROXY env variables work as expected in 0.24.x

thank you :)

@aduh95
Copy link
Contributor

aduh95 commented Apr 2, 2024

@aureq thanks for the detailed report! Can you also confirm whether you're able to reproduce with a simple fetch call outside of Corepack? If the bug is in the fetch implementation, we can write a bug report to Undici issue tracker so it can be fixed upstream.

@aureq
Copy link

aureq commented Apr 2, 2024

@aduh95 I'm trying to, but having some typing issues... Could you provide some guidance or help please?

First, I reused most of the code that's present in CorePack to save some time (the code I mentioned in my previous comment).

From the code below, vscode complains with the following error: Object literal may only specify known properties, and 'dispatcher' does not exist in type 'RequestInit'. when trying to set displatch: agent on L28 which seems to indicate the property isn't present. I'm not sure what I'm missing here.

But also, I'm a little puzzled as to why dispatcher of type Dispatcher would accept agent of type ProxyAgent` according to my code below, originally taken from https://github.com/nodejs/corepack/blob/v0.25.2/sources/httpUtils.ts#L17.

Any suggestions welcome.

async function getProxyAgent(input: string | URL) {
    const {getProxyForUrl} = await import(`proxy-from-env`);

    // @ts-expect-error - The internal implementation is compatible with a WHATWG URL instance
    const proxy = getProxyForUrl(input);

    if (!proxy) return undefined;

    // Doing a deep import here since undici isn't tree-shakeable
    const {default: ProxyAgent} = (await import(
      // @ts-expect-error No types for this specific file
      `undici/lib/proxy-agent.js`
    )) as { default: typeof import('undici').ProxyAgent };

    return new ProxyAgent(proxy);
}

async function myFetch(input: string | URL, init?: RequestInit) {
    const agent = await getProxyAgent(input);

    console.error(agent);

    let response;

    try {
        response = await globalThis.fetch(input, {
            ...init,
            dispatcher: agent, // <-- error
        });

    } catch (error) {
        console.error(error);
        throw new Error(`Error when performing the request to ${input}`);
    }
}

fetch("https://checkip.amazonaws.com");
{
    "name": "gh-417",
    "main": "index.ts",
    "engines": {
        "node": "^18.17.1 || >=20.10.0"
    },
    "devDependencies": {
        "@types/node": "^20.10.0",
        "@types/proxy-from-env": "^1",
        "proxy-from-env": "^1.1.0",
        "undici": "^6.6.1",
        "typescript": "^5.0.4",
        "ts-node": "^10.0.0"
    }
}

@aduh95
Copy link
Contributor

aduh95 commented Apr 2, 2024

But also, I'm a little puzzled as to why dispatcher of type Dispatcher would accept agent of type ProxyAgent` according to my code below

As documented in https://undici.nodejs.org/#/docs/api/ProxyAgent, ProxyAgent extends Dispatcher.

From the code below, vscode complains with the following error: Object literal may only specify known properties, and 'dispatcher' does not exist in type 'RequestInit'. when trying to set displatch: agent on L28 which seems to indicate the property isn't present. I'm not sure what I'm missing here.

That's probably because dispatcher is a Undici-specific option. You can see Undici reads the dispatcher property from the option bag either in https://github.com/nodejs/undici/blob/8dea74494cada360ae9d88a23d55b6296b9016ba/lib/web/fetch/request.js#L82 or in https://github.com/nodejs/undici/blob/8dea74494cada360ae9d88a23d55b6296b9016ba/lib/web/fetch/request.js#L107.
I think it's fine to ignore that error, but if you prefer you can also use setGlobalDispatcher(proxyAgent), but I'd expect you'd get the exact same result.
You could also try to reproduce with JS code (instead of TS), to reduce the scope further.

Does the above code reproduce the error? Could you also try to reproduce with URLs Corepack uses, e.g. https://repo.yarnpkg.com/4.0.0/packages/yarnpkg-cli/bin/yarn.js or https://registry.yarnpkg.com/yarn/latest.

@aureq
Copy link

aureq commented Apr 3, 2024

@aduh95 Thanks for the information.

It's probably worth mentioning I'm not a JS/TS developer.

With that said, I've simplified my example and the code shared below works as expected. The connection is made to my proxy and the content appears to be retrieved correctly.

So, in my example, undici appears to be working just fine. The corepack code is a little too complex for me to grasp its nuances.

But FWIW, the issue may be on corepack's side.

import * as proxyFromEnv from "proxy-from-env"
import * as proxyAgent from "undici"


async function getProxyAgent(input: string) {
    const proxy = proxyFromEnv.getProxyForUrl(input);
    if (!proxy) return undefined;
    return new proxyAgent.ProxyAgent(proxy);
}

async function myFetch(input: string, init?: RequestInit) {
    const agent = await getProxyAgent(input);

    const u = new URL(input);

    let response;

    try {
        proxyAgent.fetch
        response = await proxyAgent.fetch(u, {
            dispatcher: agent,
        });

    } catch (error) {
        console.error(error);
        throw new Error(`Error when performing the request to ${input}`);
    }

    console.log(response);
}

const res = myFetch("https://checkip.amazonaws.com");

@onigoetz
Copy link

onigoetz commented Apr 3, 2024

Hi,

I have the same issue and had a look and have a bit more information

My environment

Node 18.19.1
Corepack 0.25.2

1. Uncovering the real error

The error I got initially did not help much:

Internal Error: Error when performing the request to https://repo.yarnpkg.com/4.1.1/packages/yarnpkg-cli/bin/yarn.js; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
    at fetch (/opt/containerbase/tools/corepack/0.25.2/18.19.1/node_modules/corepack/dist/lib/corepack.cjs:22886:11)
    ...

So I added a console.log to get the original error:

original error TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11731:11)
    at async fetch (/opt/containerbase/tools/corepack/0.25.2/18.19.1/node_modules/corepack/dist/lib/corepack.cjs:22880:16)
    at async fetchUrlStream (/opt/containerbase/tools/corepack/0.25.2/18.19.1/node_modules/corepack/dist/lib/corepack.cjs:22918:20)
    ...
  cause: TypeError: this[kClient].connect is not a function
      at Client.connect (/opt/containerbase/tools/corepack/0.25.2/18.19.1/node_modules/corepack/dist/lib/corepack.cjs:13364:66)
      at socket (/opt/containerbase/tools/corepack/0.25.2/18.19.1/node_modules/corepack/dist/lib/corepack.cjs:12289:29)
      at ...
}

So what we get is TypeError: this[kClient].connect is not a function
apparently, from this line : https://github.com/nodejs/undici/blob/v6.6.2/lib/proxy-agent.js#L93

This error is internal to undici (the bundled library) but is triggered when calling globalThis.fetch

One thing I noticed is that Corepack bundles Undici 6.6.2
but Node.JS bundles version 5.28.3 : https://github.com/nodejs/node/releases/tag/v18.19.1

But from what I found next this seems to have no implication

2. A short reproduction:

{
    "devDependencies": {
        "proxy-from-env": "^1.1.0",
        "undici": "6.6.2"
    }
}
import { getProxyForUrl } from "proxy-from-env";
import ProxyAgent from "undici/lib/proxy-agent.js";
//import { fetch } from "undici/lib/fetch/index.js";
//import { ProxyAgent, fetch } from "undici";

async function getProxyAgent(input) {
    const proxy = getProxyForUrl(input);
    if (!proxy) return undefined;
    return new ProxyAgent(proxy);
}

async function myFetch(input, init) {
    const agent = await getProxyAgent(input);

    try {
        const response = await globalThis.fetch(input, {
            dispatcher: agent,
        });
        console.log(response.status);
    } catch (error) {
        console.error(error);
        throw new Error(`Error when performing the request to ${input}`);
    }
}

myFetch("https://repo.yarnpkg.com/4.1.1/packages/yarnpkg-cli/bin/yarn.js");

Using this with a proxy will yield the error we saw above:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11731:11)
    at async myFetch (file:///home/sgoetz/workspace/sq/repro/fetch.mjs:18:20) {
  cause: TypeError: this[kClient].connect is not a function

But by changing the following import, it works OK :

- import ProxyAgent from "undici/lib/proxy-agent.js";
+ import { ProxyAgent } from "undici";

My understanding is that some other file imported by undici has side effects and would get the library to work fine.

@boindil
Copy link

boindil commented Apr 4, 2024

This really needs to be fixed ... we do have the exact same error ...

@aureq
Copy link

aureq commented Apr 4, 2024

@boindil As suggested, if this is blocking you, you have 2 options.

  1. Downgrade to 0.24.x as suggested by me
  2. Patch the source as suggested by @onigoetz

And if this is a pressing issue, then feel free to submit a pull request for the team to review.

@twojnarowski
Copy link

I have the same issue:

Internal Error: Error when performing the request to https://registry.npmjs.org/pnpm; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
    at ClientRequest.<anonymous> (C:\Program Files\nodejs\node_modules\corepack\dist\lib\corepack.cjs:42195:14)

But my corepack version is 0.23.0, so downgrade to 24 is not possible, I have an even lower version.

@jsayer101
Copy link

I Upgraded a project from Node 16 to 20. Our build is made in a custom Docker image. I think that the bugged version of corepack is shipped by default in Node 20. I ended up upgradind corepack to 0.27.0

For reference:

RUN corepack enable && npm install -g corepack@0.27.0 && corepack install --global yarn@1.22.22

Works well behind our proxy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants