Using with AWS Lambda

[sajithneyo]

I think it's better to have correct documentation about how to use mongoose in an AWS Lambda environment. Especially when using useUnifiedTopology true and false. The document https://mongoosejs.com/docs/lambda.html does not feel updated.

Also, can someone shed some light on this? I used bellow settings with useUnifiedTopology true and it worked fine a few months ago (November 2020).

useUnifiedTopology: true,
useCreateIndex: true,
useNewUrlParser: true,
bufferCommands: false,
bufferMaxEntries: 0,
useFindAndModify: false,
autoIndex: false

But the traffic increased recently and now the max connections limit is hit often. Need to know what has changed since then as I did not work on that project until now.

I see a StackOverflow Q https://stackoverflow.com/questions/60257296/setting-buffercommands-to-false-not-working-in-mongoosejs that says bufferMaxEntries: 0 is now deprecated with useUnifiedTopology: true. Does that have an effect?

[vkarpov15]

@sajithneyo no, bufferMaxEntries doesn't have an effect with useUnifiedTopology: true.

You may want to try reducing poolSize: https://mongoosejs.com/docs/connections.html to avoid the max connections limit. Otherwise, please post some code samples that demonstrate how you're connecting to MongoDB from Lambda.

[sajithneyo]

Here's the code that I currently use to connect to the database (What's in the prod, staging use an oop singleton approach which will replace this in the future)

import util from 'util';
import mongoose, { ClientSession, Mongoose } from 'mongoose';
import { LambdaLog } from 'lambda-log';

let cachedDB: Mongoose | null = null;
const logger = new LambdaLog();
const LOGGER_PREFIX = 'MongoDB Connector v1';
const uriString =
    process.env.NODE_ENV === 'production'
        ? util.format('mongodb+srv://%s:%s@%s', process.env.MONGO_USER, process.env.MONGO_PASS, process.env.MONGO_IP)
        : 'mongodb://localhost:27017/?readPreference=primary&appname=MongoDB%20Compass&ssl=false';

export default class NewMongoConnector {
    public static async connectToDatabase(): Promise<void> {
        logger.info(`${LOGGER_PREFIX} - Connect function invoked`);

        if (cachedDB) {
            const hasAConnection = mongoose.connections.find((connection) => connection.readyState === 1);
            if (hasAConnection) {
                logger.info(`${LOGGER_PREFIX} - Reusing connection`);
                return;
            }
        }

        try {
            cachedDB = await mongoose.connect(uriString, {
                useUnifiedTopology: false,
                useCreateIndex: true,
                useNewUrlParser: true,
                bufferCommands: false,
                useFindAndModify: false,
                bufferMaxEntries: 0,
                // serverSelectionTimeoutMS: 0,
                // socketTimeoutMS: 0,
                dbName: process.env.CUSTOM_ENV === 'production' ? 'prod-db' : 'test-db',
                poolSize: 3,
                autoIndex: process.env.CUSTOM_ENV !== 'production',
            });
            logger.info(`${LOGGER_PREFIX} - New connection`);
        } catch (err) {
            logger.error(err);
            cachedDB = null;
        }
    }

    public static async disconnect(): Promise<void> {
        await mongoose.disconnect();
    }

    public static async startSession(): Promise<ClientSession> {
        const session: ClientSession = await mongoose.startSession();
        return session;
    }
}

mongoose.connection.on('opening', () => {
    logger.info(`${LOGGER_PREFIX} - Reconnecting... => ${mongoose.connection.readyState}`);
});

mongoose.connection.on('disconnected', () => {
    logger.error(`${LOGGER_PREFIX} - Could not connect to database`);
});

mongoose.connection.once('open', () => {
    logger.info(`${LOGGER_PREFIX} - Connected to MongoDB`);
});

mongoose.connection.on('reconnected', () => {
    logger.info(`${LOGGER_PREFIX} - MongoDB reconnected!`);
});

Here's a graph of the number of connections that's been used over a week. This application receives 15K-20K requests per day and has nearly 15 microservices.

1 - Using the config useUnifiedTopology: true
2, 3, 4, 5 - Still keeping that config and adding serverSelectionTimeoutMS and socketTimeoutMS configs to reduce the number of connections. This was done while moving less traffic services connect classes with the configs useUnifiedTopology: false, bufferCommands: false and bufferMaxEntries: 0.
6 - After making all the services to connect with useUnifiedTopology: false, bufferCommands: false and bufferMaxEntries: 0 configs.

So what I understood is, making useUnifiedTopology: true will result in spawing more connections. What's bugging me is, is this something with mongoose or mongodb nodejs driver. This behaviour caught my attention recently. Back in 2020 dec 2021 jan, this was not the case even though useUnifiedTopology is enabled.

This shows how the cluster behaved for the same amount of traffic in between 11/2020 to 03/2021

So do you have any clue why this is happening?

[vkarpov15]

Without knowing what version of Mongoose you're using, it's hard to be sure.

One detail I noticed that may be important: cachedDB = await mongoose.connect(uriString) means you're waiting for the connection to succeed before setting cachedDB, which means subsequent requests will call mongoose.connect() again. We recommend assigning before awaiting in our Lambda docs. Since Mongoose v5.10.11 this distinction shouldn't matter, but if you're running an older version you may be running into a case where the "awaiting before assigning" race condition causes connections to spike.

[sajithneyo]

Hey, thanks for the reply. I'm currently using mongoose 5.12.7