0

Is there a better / more optimal way how to fetch multiple queries from Mongo in NodeJS? I run this code in a loop for every user so I worry of performance and reliability of this code when my user base grows.

const cron = require('node-cron');
cron.schedule(process.env.SCHEDULE_TIME, async () => calculateUserHonors(), {
  scheduled: true,
  timezone: 'Europe/Prague',
});

const calculateUserHonors = async () => {
 // for loop for every user in database
let pollVotesCount, commentVotesCount, sharesCount, commentsCount, blogCount;
const pollVotesPromise = getPollVoteCount(dbClient, userId);
const commentVotesPromise = getCommentVotesCount(dbClient, userId);
const sharesPromise = getShareLinkCount(dbClient, userId);
const commentsPromise = getCommentedCount(dbClient, userId);
const blogPromise = getBlogCount(dbClient, userId);
Promise.all([pollVotesPromise, commentVotesPromise, sharesPromise, commentsPromise, blogPromise]).then((values) => {
  pollVotesCount = values[0];
  commentVotesCount = values[1];
  sharesCount = values[2];
  commentsCount = values[3];
  blogCount = values[4];
});

const getPollVoteCount = async (dbClient, userId) => dbClient.db().collection('poll_votes').find({ user: userId }).count();

const getCommentVotesCount = async (dbClient, userId) => dbClient.db().collection('comment_votes').find({ 'user.id': userId }).count();

const getShareLinkCount = async (dbClient, userId) => dbClient.db().collection('link_shares').find({ user: userId }).count();

const getCommentedCount = async (dbClient, userId) => dbClient.db().collection('comments').find({ 'user.id': userId }).count();

const getBlogCount = async (dbClient, userId) => dbClient.db().collection('items').find({ 'info.author.id': userId }).count();
3
  • 2
    mongo shell has a count({}) method in addition to the find({}) method. Is this available to you? Querying for the count, rather than all the records and then counting them, would seem to be more optimal.
    – Taplar
    Commented Aug 25, 2020 at 18:24
  • Thanks, this might help a bit. I am looking if it is possible to run all those 5 commands at once. Commented Aug 25, 2020 at 18:27
  • Well, I store the results back to the user item later. So I wonder that it may be more efficient to run these queries as a single update of five fields and retrieve the user again afterwards. Would it be faster? Alternatively I would be happy if I can run this update for all users at once, but I am not sure if mongo can handle such long running query. Commented Aug 26, 2020 at 6:10

1 Answer 1

1

You should look into aggregation pipelines: https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/

For example, to get the poll_votes:

dbClient.db().collection('poll_votes').aggregate([
    { $group: { _id: '$user', count: { $sum: 1 } } },
    { $addFields: { user: '$_id', pollVotes: '$count' } }
]);

You would get an array with poll votes for each user.

3
  • I use the aggregation but these collections are unrelated. Commented Aug 25, 2020 at 21:00
  • What do you mean unrelated? You are getting every user and then getting the poll_votes for each of them in a loop; my suggestion is a single query that gets the poll_votes for every user.
    – domenikk
    Commented Aug 26, 2020 at 9:26
  • I am getting data from 5 different collections. I see your point now - I can get the count for all users at once. This will be more efficient than the current solution where I call this count within a loop for each user separately. I need to think about behaviour when there will be tens of thousands of users and I need five such result sets. Commented Aug 26, 2020 at 9:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.