After upgrading to Prisma 3.10 from 3.9, remix site crashes on connection (Codesigning issue on M1 Macs)

[zayenz]

Bug description

In a Remix.run site I'm developing, I updated Prisma from 3.9.1 to 3.10. This made the server crash with the following message.

Downgrading to Prisma 3.9.2 worked again. Tested multiple times to switch version with consistent behaviour.

Watching Remix app in development mode...
💿 Built in 228ms
Remix App Server started at http://localhost:3000
fatal error: all goroutines are asleep - deadlock!
[1]    26432 killed     npx remix dev --debug
                                                                                                                                                                                            
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x1400001ad28)
        runtime/sema.go:56 +0x38
sync.(*WaitGroup).Wait(0x1400001ad20)
        sync/waitgroup.go:130 +0xa4
main.runService(0x1)
        github.com/evanw/esbuild/cmd/esbuild/service.go:138 +0x494
main.main()
        github.com/evanw/esbuild/cmd/esbuild/main.go:203 +0x1f0

goroutine 6 [chan receive]:
main.runService.func1(0x140000261e0, 0x1400001ad20)
        github.com/evanw/esbuild/cmd/esbuild/service.go:66 +0x3c
created by main.runService
        github.com/evanw/esbuild/cmd/esbuild/service.go:64 +0x1a8

goroutine 7 [chan receive]:
main.(*serviceType).sendRequest(0x140000261e0, {0x1029e7320, 0x14003885ce0})
        github.com/evanw/esbuild/cmd/esbuild/service.go:163 +0x104
main.runService.func2(0x140000261e0)
        github.com/evanw/esbuild/cmd/esbuild/service.go:92 +0x40
created by main.runService
        github.com/evanw/esbuild/cmd/esbuild/service.go:89 +0x2b8

Note that Prisma studio works in both versions.

How to reproduce

Unclear. Similar issues have been discussed in the Remix discord (https://discord.com/channels/770287896669978684/938793097414975528/938795339757010964)

See also #11995

Expected behavior

No response

Prisma information

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

System seems to crash on connection to the database.

Environment & setup

Computer: MacBook Pro 14 inch, M1 Max, macOS Monterey 12.2
Database: Postgresql 14.1 installed from homebrew
Node: v17.6.0
Remix: 1.2.2 (also tested 1.2.1)

Prisma Version

Non-working version

Environment variables loaded from .env
prisma                  : 3.10.0
@prisma/client          : 3.10.0
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node)
Migration Engine        : migration-engine-cli 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/migration-engine-darwin-arm64)
Introspection Engine    : introspection-core 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/introspection-engine-darwin-arm64)
Format Binary           : prisma-fmt 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/prisma-fmt-darwin-arm64)
Default Engines Hash    : 73e60b76d394f8d37d8ebd1f8918c79029f0db86
Studio                  : 0.458.0

Working version

Environment variables loaded from .env
prisma                  : 3.9.2
@prisma/client          : 3.9.2
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node)
Migration Engine        : migration-engine-cli bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/migration-engine-darwin-arm64)
Introspection Engine    : introspection-core bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/introspection-engine-darwin-arm64)
Format Binary           : prisma-fmt bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/prisma-fmt-darwin-arm64)
Default Engines Hash    : bcc2ff906db47790ee902e7bbc76d7ffb1893009
Studio                  : 0.457.0

[janpio]

What does npx remix dev --debug with Prisma under the hood?
Is there a way to create a minimal project that shows this behavior so we have an easier time reproducing?

[zayenz]

Remix builds the site, and starts an express server. The Prisma connection is managed through this code

export { PrismaClient as DbClient } from "@prisma/client";

let db: DbClient;

declare global {
    // eslint-disable-next-line no-var
    var __db: DbClient | undefined;
}

// this is needed because in development we don't want to restart
// the server with every change, but we want to make sure we don't
// create a new connection to the DB with every change either.
if (process.env.NODE_ENV === "production") {
    db = new DbClient();
    db.$connect();
} else {
    if (!global.__db) {
        global.__db = new DbClient();
        global.__db.$connect();
    }
    db = global.__db;
}

export { db };

which is form the Remix documentation (https://remix.run/docs/en/v1/tutorials/jokes#connect-to-the-database). As far as I can tell, it deadlocks when trying to connect to the database.

I have unfortunately not had the time to attempt a reproducing example, but I hope to do so. I wanted to create a bug before I had a reproduction with enough description that others could find it if they encounter the same issue.

[jacobwgillespie]

I experienced something similar though with a Next.js application - I wasn't able to put together a reproduction as my clean repo didn't crash the process, so I decided to try deleting all .prisma generated directories before running prisma generate again, and that worked!

I have a monorepo, so I located all .prisma directories with find . -name .prisma, then deleted each one.

So at least for me, the issue manifested when I already had a generated client from 3.9, and cleaning up the old version before running generate again appears to have fixed it.

[zayenz]

Thanks, that solved my issue! Wouldn't have occured to me, since otherwise the watcher I have regenerating prisma files has worked without a hitch.

[janpio]

Can you still follow which .prisma folder maybe was in an unexpected location?

[zayenz]

The repository I work in is not a monorepo yet, so for me it was the standard node_modules/.prisma location.

[janpio]

Interesting. Upgrading the package should also prisma generate again. Something must have gone wrong there. Super weird.

[jacobwgillespie]

@janpio fascinating, this is an interesting one, I was able to reproduce! For extra context, I did run prisma generate after upgrading to 3.10, which did not fix the issue until I deleted the .prisma/client directory.

So the issue is macOS code signing on M1 Macs!

Specifically, code signing breaks when replacing a binary with the contents of a new binary, for example you can reproduce this via the following:

$ cp /usr/bin/false ./example
$ ./example # works
$ cp /usr/bin/true ./example
$ ./example # killed by the kernel

By creating the example binary as a copy of the first binary, the binary is considered safe, but when replacing it with the copy of a different binary, it's considered unsafe.

I believe that's what is happening here, when upgrading from 3.9 to 3.10, the file node_modules/.prisma/client/libquery_engine-darwin-arm64.dylib.node is replaced with the 3.10 version, but since its path / inode is the same as before, it is considered no longer code signed by the macOS sandbox.

This manifests as the following log lines in Console:

default	22:54:01.120624+0000	kernel	CODE SIGNING: cs_invalid_page(0x118000000): p=70474[node] final status 0x23010204, denying page sending SIGKILL
default	22:54:01.120635+0000	kernel	CODE SIGNING: process 70474[node]: rejecting invalid page at address 0x118000000 from offset 0x0 in file "/Users/jacobwgillespie/Code/jacobwgillespie/primsa-3-10-issue/3-9-then-3-10/node_modules/.prisma/client/libquery_engine-darwin-arm64.dylib.node" (cs_mtime:1645742717.771967254 != mtime:1645742737.976117428) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default	22:54:01.120702+0000	kernel	node[70474] Corpse allowed 1 of 5
default	22:54:01.169387+0000	ReportCrash	Formulating fatal 309 report for corpse[70474] node
default	22:54:01.171430+0000	ReportCrash	no MetricKit for process node type 309 bundleId (null)
default	22:54:01.171521+0000	ReportCrash	Sending event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000032, 0x0000000118000000(\n    50,\n    4697620480\n)EXC_BAD_ACCESSSIGKILL (Code Signature Invalid)UNKNOWN_0x32 at 0x0000000118000000","incidentID":"15D796F0-C20F-41A4-92A4-69DA30481984","logwritten":0,"process":"node","responsibleApp":"iTerm2","terminationReasonExceptionCode":"0x2","terminationReasonNamespace":"CODESIGNING"}

Note the cs_mtime:1645742717.771967254 != mtime:1645742737.976117428 on the rejection, the file mtime at initial creation (v3.9) no longer matches the current mtime after upgrade (v3.10)

I think the potential fixes would be:

1. Delete the .dylib.node file in prisma generate before generating the new client (or delete the whole .prisma/client for that matter)
2. Put the Prisma version in the filename, so that on version upgrade, you're guaranteed to get a new inode for the binary (libquery_...3.10.0.dylib.node, etc)

Another benefit to (1) is it would clean up old files, I also discovered that after upgrading from 3.9 to 3.10 a leftover .prisma/client/runtime/esm directory was still present from 3.9, but that turned out to be a red herring. But if Prisma cleaned its client directory before generate, that would potentially also clean up leftovers like esm.

Credit to https://openradar.appspot.com/FB8914243 for the simple reproduction of the issue above.

[janpio]

Oh wow. Instant reopen of this issue then.

This points that we should probably add the version to the file name or put the engines into a folder named after the version.

[nikolai-cc]

Might be superfluous since the cause seems to have been found by @jacobwgillespie above, but can confirm the issue also happens with SvelteKit (so not only Remix) after updating Prisma to 3.10.

Attempting to connect after starting a dev server ends the process with SIGKILL.

I am also on M1 and I did run prisma generate after update. Removing ./node-modules/.prisma and rerunning prisma generate solved the issue.

[janpio]

If anyone is in the position to do this, a copy of the node_modules/.prisma copy before and after removing and regenerating it would be helpful for us just to make sure it is not something else than what @jacobwgillespie described.

[jacobwgillespie]

I still have a copy of my test files, here's some zips:

• Installing 3.9.x first, then 3.10.x: https://drive.google.com/file/d/1lAQbSArGgQAiGdHD5QXRqo9aGZCSoalX/view?usp=sharing
• Only installing 3.10.x: https://drive.google.com/file/d/1WOoxtjxdQrYcNjLKRsj6VtwjmX7LjJiG/view?usp=sharing

You can see the leftover esm directory in the first zip...

One of the things that lead to the finding above was that if I made a copy of the project the copy worked. As in, if I upgraded from 3.9 to 3.10, the project was broken, but if I cp -r node_modules to another directory it worked. That seemed to imply that it wasn't actually the contents of the files, but something about their metadata instead, which lead to investigating the kernel error messages.

[pantharshit00]

I did try this on my machine and wasn't able to reproduce. Looks like we will require more information here.

[jacobwgillespie]

@pantharshit00 do you have an M1 Mac? From my understanding this specifically triggers on darwin-arm64.

The steps to reproduce that work for me, if useful:

1. Create a package.json with the following:

   {
     "dependencies": {
       "@prisma/client": "3.9.x",
       "prisma": "3.9.x"
     }
   }

2. Run yarn install && yarn prisma generate
3. Try using Prisma and notice that it works
4. Modify the package.json to the following:

   {
     "dependencies": {
       "@prisma/client": "3.10.x",
       "prisma": "3.10.x"
     }
   }

5. Run yarn install && yarn prisma generate
6. Try using Prisma and notice that the kernel immediately kills the node process

[andremonteiro95]

Got the same issue when upgrading Prisma from version 3.14 to 3.15. I'm on a MacBook Pro (14-inch, 2021) with an M1 Pro chip.

[treble-snake]

Same here on M1 MacBook Pro and upgrading from version 3.14 to 3.15.
Removing node_modules/.prisma and running prisma generate again helped.

[BenEinwechter]

I experienced the same problem when upgrading to 4.0.0 on a Mac Book Pro M1 Max. Completely uninstalling prisma, @prisma/client, and removing node_modules/.prisma and then reinstalling everything and running prisma generate fixed the problem.

Here's first part of the node crash report that led me to this GitHub issue:

Process:               node [70784]
Path:                  /Users/USER/*/node
Identifier:            node
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        Exited process [70616]
Responsible:           iTerm2 [47725]
User ID:               501

Date/Time:             2022-07-04 20:04:29.8249 -0400
OS Version:            macOS 12.4 (21F79)
Report Version:        12
Anonymous UUID:        6BF0DEE2-D7C2-1131-726F-27072A3AE4AA

Sleep/Wake UUID:       2E147BFE-8172-4B9A-BE71-32032B4A0532

Time Awake Since Boot: 700000 seconds
Time Since Wake:       2987 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x000000014db90000
Exception Codes:       0x0000000000000032, 0x000000014db90000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 2 

prisma -v outputs:

Environment variables loaded from .env
prisma                  : 4.0.0
@prisma/client          : 4.0.0
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine da41d2bb3406da22087b849f0e911199ba4fbf11 (at node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node)
Migration Engine        : migration-engine-cli da41d2bb3406da22087b849f0e911199ba4fbf11 (at node_modules/@prisma/engines/migration-engine-darwin-arm64)
Introspection Engine    : introspection-core da41d2bb3406da22087b849f0e911199ba4fbf11 (at node_modules/@prisma/engines/introspection-engine-darwin-arm64)
Format Binary           : prisma-fmt da41d2bb3406da22087b849f0e911199ba4fbf11 (at node_modules/@prisma/engines/prisma-fmt-darwin-arm64)
Default Engines Hash    : da41d2bb3406da22087b849f0e911199ba4fbf11
Studio                  : 0.465.0

[janpio]

Related: #14058

[aqrln]

A fix for the codesigning problem (or rather a workaround since it seems to be a macOS issue) is included in 4.1.0 and later, so this should be resolved now. I'm going to go ahead and close this issue, please feel free to reopen or create a new issue if you still run into this.