PANIC: called Option::unwrap() on a None value in query-engine/core/src/interpreter/query_interpreters/nested_read.rs:231:50

[binary64]

Bug description

Hi, I'm suddenly getting this on our Prisma server:

Thrown with args: {}
Resolver info : { prev: undefined, key: 'me', typename: 'Query' }
Model : User
Error : PrismaClientRustPanicError2 [PrismaClientRustPanicError]:
Invalid `prisma.user.findUnique()` invocation:
  PANIC: called `Option::unwrap()` on a `None` value in query-engine/core/src/interpreter/query_interpreters/nested_read.rs:231:50
This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.

How to reproduce

1. Launch your prisma project
2. Leave it running for ~20 hours
3. See the above error

Expected behavior

As Douglas Adams wrote, "Dont panic!"

Prisma information

Using paljs and nexus

query { me { id } }

Environment & setup

• OS: Linux using Docker node:16-buster
• Database:

generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["selectRelationCount", "filterJson", "referentialActions", "nApi"]
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

• Node.js version: 16

Prisma Version

"@prisma/client": "2.28.0",
"prisma": "2.28.0",

[binary64]

Correction: The docker container was executing the prisma generate with 2.29.0 - pinning this to 2.28.0 to align with my package.json and re-running CI. Will keep you updated.

[binary64]

Update: Tried pure 2.28 and ~24 hours later it crashed again.

Will now try 2.29 and report back in a few days.

[janpio]

Can you share (or reconstruct) the Prisma Client query this would send, with the relevant model in your datamodel?

[binary64]

Hi Janpio, I will get that information asap.

It seems to be affecting just one of many(5) PrismaClient instances we are storing in an array, the one that has a connection_limit=500 - maybe too high?

In the meantime, I found a bit of extra info I should paste:

Hi Prisma Team! My Prisma Client just crashed. This is the report:

Versions

Name	Version
Node	v16.6.1
OS	debian-openssl-1.1.x
Prisma Client	2.28.0
Query Engine	0.1.0
Database	mysql

Query

{"X":{}}

Logs

y starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true
prisma:client:libraryEngine graphQLToJSError
prisma:client:fetcher Error: called `Option::unwrap()` on a `None` value
    at LibraryEngine.prismaGraphQLToJSError (/usr/app/node_modules/@prisma/client/runtime/index.js:23770:14)
    at LibraryEngine.request (/usr/app/node_modules/@prisma/client/runtime/index.js:23785:24)
    at async cb (/usr/app/node_modules/@prisma/client/runtime/index.js:34770:26)
    at async middleware (/usr/app/node_modules/graphql-shield/dist/generator.js:30:24)
prisma:client:libraryEngine sending request, this.libraryStarted: true
prisma:client:libraryEngine library already starting, this.libraryStarted: true

[janpio]

The connection limit should not really be a problem.

You are using the nApi preview feature right now. Does the problem also occur with that once disabled? (Also should make no difference, but who knows... it definitely is very present in the logs you posted)

[dpetrick]

The error indicates that your schema and your database may be drifting. Can you:
a) Post your datamodel.
b) Look at your MySQL tables and, if possible, post the structure here as well?

[binary64]

I wrote this healthcheck which correctly identifies when it has crashed:

if [ "$(curl -s http://localhost:4000/graphql -X POST -H 'content-type: application/json' --data-raw $'{"variables":{"username":"x", "password":""},"query":"mutation login($username: String!, $password: String!) { login(username:$username, password:$password) }"}' | jq -r '.errors[0].message')" == "Incorrect email or password" ]; then (echo PASS && exit 0); else (echo FAIL && exit 1); fi

And the model for this is:

export const Authentication = extendType({
  type: 'Mutation',
  definition(t) {
    t.field('login', {
      type: 'String',
      args: {
        username: nonNull(stringArg()),
        password: nonNull(stringArg()),
      },
      async resolve(_root, args: LoginInputDto, ctx: Context) {
        return <implementation details>
      },
    })

The "implementation details" basically calls this:

await prisma.user.findMany({
      where: {
        username: {
          equals: username,
        },
      },
      select: {
        id: true,
        username: true,
        password: true,
        password_algor: true,
        salt: true,
        hash: true,
        admin: true,
        locale: true,
        Company: {
          select: {
            id: true,
            admin: true,

and the model for user is

model User {
  id                           Int                            @id @default(autoincrement())
  username                     String?
  full_name                    String
<snipped, as there is 100s of lines here>
  CompanyOwner                 Company?                       @relation("owner")
  PasswordResetAuth PasswordResetAuth[]
  @@unique([id, company_id], name: "primary")
  @@index([calendar_order], name: "calendar_order")
  @@index([company_id], name: "company")
  @@index([deleted], name: "deleted")
  @@index([email], name: "email")
  @@index([full_name], name: "full_name")
  @@index([hash], name: "hash_index")
  @@index([username], name: "username")
  @@map("users")
}

I could get the mysql schema too, but why would it work for many hours then die. I would expect a schema mis-match to not work straight away, and also be citing MySQL error codes

Our complete schema is propreitary however we are willing to send this as a confidential email to you.

Thank for your help with this. I will disable nApi now and report back in a day.

[binary64]

From what I can tell, 2.29 running without nApi is running solidly so far.

[janpio]

Although we can not reproduce this, we can not imagine how this could be connected to using the nApi preview feature or not. Our current theory is that some data is corrupted once during for some weird reason, which is making the Engine crash internally.

If you get any additional information on how to reliably reproduce this, or can indeed pinpoint this being connected to our Node-API engine exclusively, we are super happy to jump on this again. For now we do not really know how to proceed further here.

[binary64]

Can I DM you on Slack with a private repo?

Some extra weak information I have:

• My belief is the problem started after 2.27
• We use nexus with the paljs plugin
• We use some object and json fields

I also do not know how to proceed. My $0.02 for the Prisma team is: As the error I hit is the same file and line number each time, could you catch that particular error and output to console any extra information you guys need me to paste here? Like a core dump. Maybe only when I have a certain env var set (otherwise this might be adding an extra vector for attackers).

[janpio]

Yes, I am @janpio on our public Slack as well - just ping me there.

As the error I hit is the same file and line number each time, could you catch that particular error and output to console any extra information you guys need me to paste here? Like a core dump. Maybe only when I have a certain env var set (otherwise this might be adding an extra vector for attackers).

We could make the crash disappear indeed - but are hesitant as it really might be a deeper problem that needs to be solved.

[binary64]

Update - just had another PANIC of the exact same signature. So we can now rule out nApi being the cause.

[elie222]

Also ran into this issue today. It happens randomly. Not sure how to reproduce. Possibly by making too many requests to Prisma at once.

[janpio]

@elie222 opened a separate issue about this as well: #10049 Their error message is a bit different, so that makes sense.

[binary64]

Update, I have got this crash again on 1 of my servers in a cluster. The rest of the cluster is fine, it's just 1 node that seems to crash with this anytime we call prisma:-

PANIC: called `Option::unwrap()` on a `None` value in query-engine/core/src/interpreter/query_interpreters/nested_read.rs:232:50\n\nThis is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.

Notice how the line number has increased by 1 so I assume it's the same problem here that I've had for a year or so so far.

[elie222]

I can confirm the PANIC error happens with nested operations.
As a short term (and what worked for us), try to unnest your create/update.
Either as a $transaction or multiple db updates.

[binary64]

We are still experiencing this issue with the latest version 3.10.0

[binary64]

I can confirm the PANIC error happens with nested operations. As a short term (and what worked for us), try to unnest your create/update. Either as a $transaction or multiple db updates.

We are seeing this with query{me{id}} as well

[binary64]

My hunches are:

• In our express server, we have a global array of PrismaClients. Each customer has their own mysql database. Plus we have one master mysql which has the list of the customers. So in some resolvers, we even call both the master, and the customer PrismaClient.
• There is a memory issue where you do a large query that somehow starves or corrupts the arguments for the nested_join.rs function until the node is restarted.

[andrewicarlson]

I can reproduce this issue. Please see this repo for a small reproduction. It seems as though this Rust Panic occurs when a relationship is created using a non-unique constraint. Some connectors check for this (SQLite) but others don't and cause a runtime error (MySQL)

Notes:

1. When using the SQLite connector an error is thrown in the command line tooling, catching this error before it gets to production. MySQL throws no error and then the Rust engine Panics at runtime.

Error: P1012

error: Error validating: The argument `references` must refer to a unique criteria in the related model `A`. But it is referencing the following fields that are not a unique criteria: custom_id
  -->  schema.prisma:21

2. This reproduction uses v3.11.0. In >v3.12 an opaque error is thrown in the command line tooling that is difficult to debug.

An error occured while running the seed command:
Error: Command was killed with SIGKILL (Forced termination): ts-node prisma/seed.ts

[janpio]

I can reproduce the "called Option::unwrap() on a None value" problem with the reproduction you provided, also that the SQLite provider has additional validations that somehow got lost for MySQL.

I can not reproduce the Error: Command was killed with SIGKILL (Forced termination): ts-node prisma/seed.ts in newer Node versions.

Considering the long history of this issue, I suggest I create a new one with a minimal reproduction and nothing else.

[janpio]

New issue: #13340 I slightly simplified the reproduction from you @andrewicarlson. I also already added that this is probably a missing validation in the mysql provider (it also does not validate for postgresql) which makes this a problem of schema validation - and the Query Engine and Prisma Client should never have tried to run a query with this database schema.

Please confirm that this actually covers your cases and let me know if not.

[andrewicarlson]

I can not reproduce the Error: Command was killed with SIGKILL (Forced termination): ts-node prisma/seed.ts in newer Node versions.

Just checking, did you mean newer Prisma versions?

Please confirm that this actually covers your cases and let me know if not.

I think handling this via schema validation makes sense and would allow this issue to be caught and resolved pre-runtime more easily.

[janpio]

Just checking, did you mean newer Prisma versions?

Yes indeed, sorry.