Mongodb introspection fails with "called Option::unwrap() on a None value"

[alavkx]

Happy to provide more info. Very new to mongo so I'll need some direction to be of any use.

Command

db pull

Versions

Name	Version
Platform	darwin-arm64
Node	v16.15.0
Prisma CLI	4.0.0
Engine	da41d2bb3406da22087b849f0e911199ba4fbf11

Error

Error: [introspection-engine/connectors/mongodb-introspection-connector/src/sampler/statistics/indices.rs:222:67] called `Option::unwrap()` on a `None` value

[pimeys]

Do you have an index such as:

{ "a.b.c": 1 }

defined in your database, but then in your data b is not an embedded type but a scalar? We cannot decipher any working data model for this, but at least we should not crash.

[alavkx]

I'm brand new to this codebase, and to mongo. Do you know a way I grab a list of all the indexes in the db? I wrote a script last week to get a list of keys for every collection, but I think indexes are something different..?

edit: I'll try something like this and see what I get.

var map={}; 
db.getCollectionNames().forEach(function(coll){
  db.getCollection(coll).getIndexes().forEach(function (i){
    map[coll]=i
  })
});
map;

edit2: Here is the map of indexes in the database keyed by collection name. Is this helpful?

{
  dataexports: { v: 2, key: { type: 1 }, name: 'type_1' },
  inventorybanners: { v: 2, key: { organization: 1 }, name: 'organization_1' },
  deliveryzones: { v: 2, key: { venue: 1 }, name: 'venue_1' },
  venues: { v: 2, key: { 'default.created': 1 }, name: 'default.created_1' },
  orderstats: {
    v: 2,
    key: { 'totalsByAgeGroup.ageGroup6.created': 1 },
    name: 'totalsByAgeGroup.ageGroup6.created_1',
    background: true,
    expireAfterSeconds: 120
  },
  venuerewards: {
    v: 2,
    key: { 'alpineIQ.id': 1 },
    name: 'alpineIQ.id_1',
    background: true,
    sparse: true
  },
  prospects: { v: 1, key: { created: 1 }, name: 'created_1' },
  productreviews: { v: 2, key: { customer: 1 }, name: 'customer_1', sparse: true },
  recentpurchases: {
    v: 2,
    key: { venue: 1, product: 1 },
    name: 'venue_1_product_1',
    background: true
  },
  organizations: { v: 1, key: { billingStatus: 1 }, name: 'billingStatus_1' },
  daysettings: { v: 2, key: { venue: 1 }, name: 'venue_1' },
  inventorycategories: { v: 2, key: { venue: 1, order: 1 }, name: 'venue_1_order_1' },
  venuemetrics: {
    v: 2,
    key: { date: 1, type: 1, period: 1 },
    name: 'date_1_type_1_period_1',
    background: true
  },
  apipartnerrequestlogs: { v: 2, key: { apiPartner: 1 }, name: 'apiPartner_1' },
  daystats: { v: 2, key: { created: 1 }, name: 'created_1' },
  ordercounters: { v: 2, key: { created: 1 }, name: 'created_1' },
  products: {
    v: 2,
    key: { venue: 1, 'leaflogix.productId': 1, quantity: 1 },
    name: 'venue_1_leaflogix.productId_1_quantity_1',
    sparse: true
  },
  customers: {
    v: 2,
    key: {
      organization: 1,
      venues: 1,
      marketingOptIn: 1,
      deleted: 1,
      marketingOptInDate: 1
    },
    name: 'organization_1_venues_1_marketingOptIn_1_deleted_1_marketingOptInDate_1',
    background: true
  },
  productbundles: { v: 2, key: { venue: 1 }, name: 'venue_1' },
  managementusers: { v: 1, key: { apiKey: 1 }, name: 'apiKey_1', sparse: true },
  tickets: { v: 2, key: { order: 1 }, name: 'order_1', sparse: true },
  productmetrics: { v: 2, key: { product: 1 }, name: 'product_1' },
  notificationsettings: {
    v: 2,
    key: { type: 1, venue: 1, created: -1 },
    name: 'type_1_venue_1_created_-1'
  },
  printerjobs: { v: 2, key: { printer: 1 }, name: 'printer_1' },
  tracks: {
    v: 2,
    key: { admissionGroup: 1 },
    name: 'admissionGroup_1',
    sparse: true
  },
  venuemetricbatches: { v: 2, key: { date: 1 }, name: 'date_1' },
  venueoffers: {
    v: 2,
    key: { venue: 1, 'springbig.id': 1, springbig: 1 },
    name: 'venue_1_springbig.id_1_springbig_1',
    background: true
  },
  apipartners: { v: 2, key: { apiKey: 1 }, name: 'apiKey_1', unique: true },
  ordercarts: {
    v: 2,
    key: { 'discounts.alpineIQ.id': 1 },
    name: 'discounts.alpineIQ.id_1',
    background: true,
    sparse: true
  },
  organizationpermissions: {
    v: 1,
    key: { managementUser: 1, venue: 1 },
    name: 'managementUser_1_venue_1'
  },
  productdeals: { v: 2, key: { venue: 1, order: 1 }, name: 'venue_1_order_1' },
  organizationmembers: {
    v: 1,
    key: { organization: 1, managementUser: 1 },
    name: 'organization_1_managementUser_1',
    unique: true
  },
  inventorystats: { v: 2, key: { created: 1 }, name: 'created_1' },
  authcredentials: { v: 1, key: { deleted: 1 }, name: 'deleted_1', sparse: true },
  inventorybrands: { v: 2, key: { venue: 1 }, name: 'venue_1' },
  printers: { v: 2, key: { venue: 1 }, name: 'venue_1' },
  inventories: {
    v: 2,
    key: {
      venue: 1,
      'leaflogix.productId': 1,
      'leaflogix.roomId': 1,
      date: 1
    },
    name: 'venue_1_leaflogix.productId_1_leaflogix.roomId_1_date_1',
    sparse: true
  },
  users: { v: 2, key: { email: 1 }, name: 'email_1', sparse: true },
  customerchatmessages: {
    v: 2,
    key: { deleted: 1, venue: 1, customer: 1, visibility: 1, type: 1 },
    name: 'deleted_1_venue_1_customer_1_visibility_1_type_1',
    background: true
  },
  reports: {
    v: 2,
    key: { 'default.organization': 1 },
    name: 'default.organization_1',
    background: true,
    sparse: true
  },
  customerchatstats: {
    v: 2,
    key: { created: 1 },
    name: 'created_1',
    background: true,
    expireAfterSeconds: 120
  },
  customerchatthreads: {
    v: 2,
    key: { organization: 1, status: 1, venue: 1, deleted: 1 },
    name: 'organization_1_status_1_venue_1_deleted_1',
    background: true
  },
  organizationdataexports: { v: 2, key: { _id: 1 }, name: '_id_' },
  ticketfiles: { v: 2, key: { created: 1 }, name: 'created_1' },
  organizationmetrics: { v: 2, key: { period: 1 }, name: 'period_1' },
  auths: {
    v: 1,
    key: { managementUser: 1 },
    name: 'managementUser_1',
    sparse: true
  },
  orders: {
    v: 2,
    key: { organization: 1, type: 1 },
    name: 'organization_1_type_1',
    background: true
  },
  organizationintegrationlogs: {
    v: 2,
    key: { organizationIntegration: 1 },
    name: 'organizationIntegration_1'
  },
  customersessions: {
    v: 2,
    key: { organization: 1, venue: 1, created: 1, deleted: 1, prospect: 1 },
    name: 'organization_1_venue_1_created_1_deleted_1_prospect_1',
    background: true
  },
  organizationintegrations: { v: 2, key: { created: 1 }, name: 'created_1' },
  permissions: { v: 2, key: { _id: 1 }, name: '_id_' }
}

[pimeys]

So you should check in the data if any of the following fields points to an embedded type in every single document:

• venues.default
• orderstats.totalsByAgeGroup
• orderstats.totalsByAgeGroup.ageGroup6
• venuerewards.alpineIQ
• products.leaflogix
• venueoffers.springbig
• ordercarts.discoungs
• ordercarts.discounts.alpineIQ
• inventories.leaflogix
• reports.default

I'm pushing a fix that does not crash here anymore, but we cannot really generate an immediately working data model if the data is super dirty... Of course it's easier to fix if you at least get some data model out...

[pimeys]

The result would be something like this https://github.com/prisma/prisma-engines/pull/3039/files#diff-a4d0b8785181a209f54dbbdcd39d6c0168a7c6431c2462c043c8f25456f2c3b7R1989-R2030

[alavkx]

Is there a straightforward way to point npx to a dev branch? I can run again with your change-set and let you know the result.

[pimeys]

Yeah when a dev is out. The PR is still not approved by anybody and I'm not sure is this the solution we choose. I'll mark the dev release here when one is out.

[pimeys]

This nice GitHub feature shows the release in the PR where the fix is available: prisma/prisma-engines#3039

(spoiler: npm i prisma@4.1.0-dev.40)

[alavkx]

Ah saw the merge but didn't know how to tie that to a version number. Cheers I'll try that now.

[alavkx]

yarn add -D prisma@4.1.0-dev.40 && yarn prisma db pull worked!!! U are the mans ᕙ༼ຈل͜ຈ༽ᕗ thank you so much ❤️ ‼️

[pimeys]

Did the resulting data model make any sense?

[alavkx]

We have a giant data model, 4300 lines. I'd need someone more experienced with the model to review it all, but it generally looks really solid! I'm working through caught issues with the model to get prisma generate working 👍
The errors types I've seen so far

• Field "x" is already defined on composite type "y".
• Error validating model "x": The index definition refers to the unknown fields: ys