Aggregated subdocuments is not populated with Model.populate(docs, opts).

[EmilsWebbod]

Do you want to request a feature or report a bug?
Bug / feature

What is the current behavior?
Aggregated subdocuments is not populated with Model.populate(docs, opts).

Using aggregate to query subdocuments and want to populate data after its been fetched.
If this change is added I can use the same function to populate post fetch on single and multiple documents.

My main code seems to query the wrong model with this setup. used debug: true
schema: user: { type: Schema.Types.ObjectId, ref: 'OrganizationUser' }
OrganizationUser.populate(docs, { path: 'user' }) -> users.find({ _id: { '$in': [ new ObjectId("611a8052425e485f982e14c7") ], [Symbol(mongoose#trustedSymbol)]: true }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, projection: { _id: 1 }})

If the current behavior is a bug, please provide the steps to reproduce.

This code will not query and models with .populate as with my main code.

mongoose.connect('mongodb://localhost:27017/testing', {}, () => {
  console.info(`Connected successfully to DB`);
});
mongoose.set('debug', true);

const UserSchema = new Schema({
  name: { type: String, default: '' },
});

const TestSchema = new Schema({
  users: {
    type: [
      {
        user: { type: Schema.Types.ObjectId, ref: 'User' },
      },
    ],
  },
});

const User = mongoose.model('User', UserSchema);
const Test = mongoose.model('Test', TestSchema);

(async function () {
  // // init data on first run
  // await User.deleteMany({});
  // await Test.deleteMany({});
  // const users = await User.create([{ name: 'user-name' }, { name: 'user-name-2' }]);
  // await Test.create([{ users: [{ user: users[0]._id }, { user: users[1]._id }] }]);

  const found: any = await Test.aggregate([
    {
      $project: {
        users: '$users',
      },
    },
    { $unwind: `$users` },
  ]);

  const userObject = found.reduce(
    (obj: any, x: any) => ({
      count: obj.count + 1,
      users: [...obj.users, x.users],
    }),
    { users: [], count: 0 } as any
  );

  await User.populate(userObject.users, { path: 'user' });
  console.log(userObject);
// {
//   count: 2,
//   users: [
//     {
//       user: new ObjectId("6193e714aed59d42d2bd0b17"),
//       _id: new ObjectId("6193e714aed59d42d2bd0b1c")
//     },
//     {
//       user: new ObjectId("6193e714aed59d42d2bd0b18"),
//       _id: new ObjectId("6193e714aed59d42d2bd0b1d")
//     }
//   ]
// }

})();

{
  "include": ["lib/**/*"],
  "exclude": ["node_modules"],
  "compilerOptions": {
    "lib": ["dom", "es2019"],
    "module": "commonjs",
    "target": "es6",
    "outDir": "./dist",
    "rootDir": "lib",
    "baseUrl": ".",
    "sourceMap": false,
    "noUnusedLocals": false,
    "noImplicitAny": true,
    "moduleResolution": "node",
    "removeComments": true,
    "resolveJsonModule": true,
    "skipLibCheck": true,
    "allowJs": true,
    "strict": true,
    "jsx": "react"
  }
}

What is the expected behavior?
Document or documents should be populated with data with the provided Model used for population.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.
Node.js 14.16.1
MongoDB 4.4.4
Mongoose: 6.0.13

[gramliu]

There are three problems here. First, you're calling populate on the User class instead of the Test class. From the documentation,

Changed in Mongoose 6: the model you call populate() on should be the "local field" model, not the "foreign field" model.

Second, you don't seem to be printing the actual result of your aggregation. Calling populate doesn't modify the original object. Instead, it returns a populated object. Running the following code should achieve what you want to do:

  const found = await Test.aggregate([
    {
      $project: {
        users: '$users',
      },
    },
    { $unwind: `$users` },
  ]);

  console.log("Found", found);

  const populated = await Test.populate(found, { path: 'users.user' });
  console.log(util.inspect(populated, false, null, true))

The last problem is that you're not passing the aggregation result into the population directly. Instead, you're mapping the aggregation results into another array. I'm not sure why this prevents the aggregation from working, probably because it doesn't retain any schema information anymore, however, moving it to after the population works as intended for me.

const found = await Test.aggregate([
    {
      $project: {
        users: '$users',
      },
    },
    { $unwind: `$users` },
  ]);

  console.log("Found", found);

  const populated = await Test.populate(found, { path: 'users.user' });

  const userObject = found.reduce(
    (obj, x) => ({
      count: obj.count + 1,
      users: [...obj.users, x.users],
    }),
    { users: [], count: 0 }
  );
  console.log(util.inspect(userObject, false, null, true))

[EmilsWebbod]

Thanks for the help. I tried moving my population right after the aggregation instead, before the mapping. This seems to not have worked for my case. I'll try some more to find a way around, and add more comment to this post. I always find a way, just hoping this was an actual bug to keep my code clean.
I'll add some more of my thought under. Thanks for the time you guys put in to this project, it helps allot!

Comment to next line, this seems that it should be the case. But all my other test seems to modify the documents directly also.

Second, you don't seem to be printing the actual result of your aggregation. Calling populate doesn't modify the original object. 
Instead, it returns a populated object. Running the following code should achieve what you want to do:

Back to my issue, thought it was strange that this worked in any other case I was using it. That's why I added it as a bug/feature

Been using mongoose for a while and always had to use the Model I wanted to populate the data with. Think this was the case when I wanted to populate the data 2-3 layers down in arrays and only wanted to populate on item instead of them all on find().populate(..)

example of a Model I have

categories: [
  sections: [
    products: [
      { product: { ref: 'Product' }, producer: { ref: 'Producer' }
    ]
  ]
]

I have some logic that would filter products before populating the data. The list can be up to 100 different products, so to reduce data fetched from DB I did all the logic then only populate the category I want to get out.

To get this to work I had to use something like this

const doc = await List.findOne({...)).lean()
const logic = doc.categories.reduce(toLogicData))
doc.categories = doc.categories.map(doSomeLogic(logic)).filter(empty)
const category = doc.categories[0]

await Promise.all([
  Product.populate(category , { path: 'categories.sections.products.product' }),
  Producer.populate(category , { path: 'categories.sections.products.producer' })
])

return category;

I do allot of mapping in the logic and copy of items with {...}, so pretty sure the schema information connected to the document is no longer there after I'm done.

If I tried using the List model it would not populate or give me null in the product or producer fields.

List.populate(category, [
 { path: 'categories.sections.products.product' },
 { path: 'categories.sections.products.producer' }
])

This may have been fixed in 6.x but I'm still running "5.13.8". Tried to upgrade, but the new changes in versions after has some major changes to TS for my code and need to get the time to upgrade.

Would upgrading to 6 break my way of populating? Product.populate(category , { path: 'categories.sections.products.product' }),

Changed in Mongoose 6: the model you call populate() on should be the "local field" model, not the "foreign field" model.

---

I'll add more to my original problems and what code caused my issue.
This is my original code, not sure if it makes sense standalone. Trying to connect query in url to population. example (populate=user:_id%20name). So server has less code and I can choose myself what I want to populate with some logic in backend. Tried to use the already defined Ref in Schema model to populate the data I wanted. This worked in almost all cases except the aggregate + reduce.

private populateDoc<D extends Document>(
    doc: D, // doc or subdoc of models
    populates: ModelSearchPopulate<D, keyof D>[],
    subPath?: string
  ) {
    return Promise.all(
      populates.map(async populate => {
        try {
          let subPathDoc: any;
          if (subPath) {
            // @ts-ignore subPaths is there. Not typed
            subPathDoc = this._model.schema.subpaths[
              `${subPath}.${populate.path}`
            ];
          } else {
            subPathDoc = this._model.schema.paths[populate.path];
          }
          let ref = subPathDoc.options.ref;
          if (!ref && Array.isArray(subPathDoc.options.type)) {
            ref = subPathDoc.options.type[0].ref;
          }
          if (!ref) {
            throw new Error(
              `ref was not defined in model with path ${populate.path}`
            );
          }
          const model = mongoose.model(ref);
          await model.populate(doc, {
            path: populate.path,
            select: populate.select.join(' ')
          });
        } catch (e) {
          console.error(e);
          console.error({ doc, populates, path: subPath });
          return Promise.resolve();
        }
      })
    );

My ModeSearchPopulate whould look something like this. Using TS with keyof T to type valid fields and subfields to populate.

{
 path: 'user',
 select: ['_id', 'name'] // reduce population select fields from frontend
 subs: [
  { sub: 'users', populate: [{ path: 'user', select: ['_id', 'name'] }] }
 ]
}

[vkarpov15]

Fix will be in 6.0.14. As a workaround in 6.0.13, you can do await User.populate(userObject.users, { path: 'user', model: User });