gatsby-source-mongodb not fetching Atlas data

[judsonlmoore]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

The gatsby-source-mongodb plugin seems not to be working with MongoDB Atlas. That, or I'm just very bad at understanding the documentation (in which case, maybe the documentation needs an explicit config example for Atlas users.

The current instructions for connectionString lead me to believe that I need to ignore some of the other parameters which are shown in the code examples, but with or without various combinations of my connectionString and the other parameters, I'm not able to establish a connection.

connectionString: if you need to use a connection string compatible with later versions of MongoDB, or for connections to MongoDB Atlas, you can enter the entire string, minus the dbName and extraParams. In this case, the authentication information should already be embedded in the string ex. mongodb+srv://<USERNAME>:<PASSWORD>@<SERVERNANE>-fsokc.mongodb.net. Pass dbName and extraParams as options per below.

I've verified with an 11ty build using npm mongodb and with MongoDB Compass on this computer that the connection string I am providing is correct.

Mentioning @pieh here since I saw you on recent commits related to this part of the repo. 🙏

Reproduction Link

https://github.com/judsonlmoore/gatsby-temp

Steps to Reproduce

1. Build a Gatsby Site following the Getting Started Tutorial
2. Install gatsby-source-mongodb following the instructions found here
3. Use any one of these configurations in the gatsby-config.js file and see the Actual Result:

[
  {
    resolve: 'gatsby-source-mongodb',
    options: {
      connectionString: 'mongodb+srv://<USERNAME>:<PASSWORD>@twopointoh.0lxmxaq.mongodb.net',
      dbName: 'test',
      collection: 'products',
      server: {
        address: '0lxmxaq.mongodb.net',
        port: 27017
      },
      extraParams: {
        replicaSet: '0lxmxaq',
        ssl: true,
        authSource: 'admin',
        retryWrites: true,
        w: `majority`,
      }
    }
  },
  {
    resolve: 'gatsby-source-mongodb',
    options: {
      connectionString: 'mongodb+srv://<USERNAME>:<PASSWORD>@twopointoh.0lxmxaq.mongodb.net',
      dbName: 'test',
      collection: 'products',
      extraParams: {
        retryWrites: true,
        w: `majority`,
      }
    }
  },
  {
    resolve: 'gatsby-source-mongodb',
    options: {
      connectionString: 'mongodb+srv://<USERNAME>:<PASSWORD>@twopointoh.0lxmxaq.mongodb.net',
      dbName: 'test',
    }
  }
]

Expected Result

MongoDB data is available in graphql.

Actual Result

Depending on how I try to setup the plugin, I get any combination of these messages:

warn MongoServerSelectionError: getaddrinfo ENOTFOUND twopointoh.0lxmxaq.mongodb.net
    at Timeout._onTimeout (/Users/judsonlmoore/GitHub/bikefi-nextjs/gatsby/node_modules/mongodb/lib/core/sdam/topology.js:438:30)
    at listOnTimeout (node:internal/timers:569:17)
    at processTimers (node:internal/timers:512:7) {
  reason: TopologyDescription {
    type: 'ReplicaSetNoPrimary',
    setName: null,
    maxSetVersion: null,
    maxElectionId: null,
    servers: Map(1) {
      'twopointoh.0lxmxaq.mongodb.net:27017' => [ServerDescription]
    },
    stale: false,
    compatible: true,
    compatibilityError: null,
    logicalSessionTimeoutMinutes: null,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    commonWireVersion: null
  }
}

or

warn The gatsby-source-mongodb plugin has generated no Gatsby nodes. Do you need it? This could also suggest the plugin is misconfigured.

or build process gets stuck on this message

⠦ source and transform nodes

Environment

System:
    OS: macOS 14.2.1
    CPU: (8) arm64 Apple M2
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.18.0 - ~/.nodenv/versions/18.18.0/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.8.1 - ~/.nodenv/versions/18.18.0/bin/npm
  Browsers:
    Chrome: 121.0.6167.139
    Safari: 17.2.1
  npmPackages:
    gatsby: ^5.13.2 => 5.13.2
    gatsby-plugin-image: ^3.13.1 => 3.13.1
    gatsby-plugin-mdx: ^5.13.1 => 5.13.1
    gatsby-plugin-sharp: ^5.13.1 => 5.13.1
    gatsby-source-filesystem: ^5.13.1 => 5.13.1
    gatsby-source-mongodb: ^5.13.1 => 5.13.1
    gatsby-transformer-sharp: ^5.13.1 => 5.13.1
  npmGlobalPackages:
    gatsby-cli: 5.13.2

Config Flags

No response

[judsonlmoore]

After sleeping on it, I thought to try again with a fresh mongodb cluster and using mongodb example data (sample_mflix dataset). With this, I was able to make a connection using these settings:

    {
      resolve: `gatsby-source-mongodb`,
      options: {
        connectionString: process.env.MONGODB_URI,
        dbName: process.env.MONGODB_DB,
        collection: process.env.MONGODB_COLLECTION,
        extraParams: {
          retryWrites: true,
          w: `majority`,
        },
      },
    },

So having established a connect to mongodb, I tried again with my data and again it failed. So then I got to thinking my data is too big (±300k items). Turns out, this is likely the problem. When I adjusted my settings to include a query to limit what data I got back, I started getting data:

    {
      resolve: `gatsby-source-mongodb`,
      options: {
        connectionString: process.env.MONGODB_URI,
        dbName: process.env.MONGODB_DB,
        collection: process.env.MONGODB_COLLECTION,
        extraParams: {
          retryWrites: true,
          w: `majority`,
        },
        query: {
          [process.env.MONGODB_COLLECTION]: { field: { $eq: value } }
        },
      },
    },

It appears that the plugin does not support mongodb aggregations, which would be better than the query method (if that's wrong, documentation about using aggregation with this plugin would be great).

Additionally, how can a site using this plugin scale to hundreds of thousands / millions of pages if indeed the plugin struggles to pull so much data? Is there another method? Further optimizations?

So at this point I'm not 100% sure if this is a bug, or a support request, but I'll leave this here for now.

[judsonlmoore]

Update: I reported this issue while traveling and working with my Apple MBA M2 laptop, now that I am home, I'm trying to continue development on an Apple iMac i7 computer (specs below from gatsby info --clipboard) and with the exact same settings and .env data, I am getting this error in the console:

ERROR  UNKNOWN
(node:6004) [DEP0170] DeprecationWarning: The URL mongodb://{username}:{password}@ac-lahiygk-shard-00-01
.0lxmxaq.mongodb.net:27017,ac-lahiygk-shard-00-00.0lxmxaq.mongodb.net:27017,ac-lahiygk-shard-00-02.0lxmxaq.mongodb.n
et:27017/test?authSource=admin&replicaSet=atlas-3ei0ma-shard-0&retryWrites=true&w=majority&ssl=true is invalid.
Future versions of Node.js will throw an error.
(Use `node --trace-deprecation ...` to show where the warning was created)

Which is really weird because a) I don't get it on both computers, and b) the URL in that error isn't constructed similarly to the one I am providing: mongodb+srv://{username}:{password}@{database}.0lxmxaq.mongodb.net

So I have no idea where that error is getting that URL from.

System:
    OS: macOS 13.6.3
    CPU: (8) x64 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.21 - /usr/local/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm
  Browsers:
    Chrome: 121.0.6167.139
    Safari: 17.1.2
  npmPackages:
    gatsby: ^5.13.2 => 5.13.3
    gatsby-plugin-image: ^3.13.1 => 3.13.1
    gatsby-plugin-mdx: ^5.13.1 => 5.13.1
    gatsby-plugin-postcss: ^6.13.1 => 6.13.1
    gatsby-plugin-sharp: ^5.13.1 => 5.13.1
    gatsby-plugin-sitemap: ^6.13.1 => 6.13.1
    gatsby-source-filesystem: ^5.13.1 => 5.13.1
    gatsby-source-mongodb: ^5.13.1 => 5.13.1
    gatsby-transformer-sharp: ^5.13.1 => 5.13.1

[judsonlmoore]

Update: I noticed that on the first computer I was running node v18.18.0 and on the second computer I was running node v20.10.0. I changed the node version on the second computer to match the first and now I get a successful build.

[apatel369]

@judsonlmoore what version of MongoDB have you used in MongoDB Atlas?

[judsonlmoore]

@apatel369 MongoDB Atlas v7.0.8 operating in region AWS N. Virginia (us-east-1) on an M0 cluster.