Nodejs function running every 2 minutes lost over time

Question

This is quite hard problem to describe. I have a koajs app with a function which is created in multiple instances (10-1000 range) every 2 minutes. this scheduled job created on app startup. I use koajs because i need a few simple api endpoints for this app. It is running well for first 3-5 hours and then the count of created instances starts to decrease and some of the log output disappears.

Here is the minimal sample based on actual code:

server.ts

const bootstrap = async () => {
    process.setMaxListeners(0); //(node:7310) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
                                //appears on app startup (however seems like this setMaxListeners(0) doesnt affect anything since the warning persist)
    const app = new Koa();
    app.use(async ctx => {
        ctx.body = "Welcome to my Server!";
    });
    app.listen(port);
    new Main().run();
};

bootstrap(); 

main.ts (tried: cron npm package, node-scheduler, setInterval, recursive setTimeout) to run the scheduledJobWrapper.

    isStarting: boolean = false;

    async run() {
        logger.info(`running the app, every 2 minutes`);

        //let that = this;
        // new CronJob(`*/2 * * * *`, function () {
        //     that.scheduledJobWrapper();
        // }, null, true, 'America/Los_Angeles');

        const interval = 2 * 60 * 1000;
        setInterval(() => {
            this.scheduledJobWrapper();
        }, interval);
    }

    async scheduledJobWrapper() {
        logger.info("here scheduledJobWrapper");

        let args = {};
        //some irrelevant logic to set the arguments
        await this.scheduledJob(args);
    }

    async scheduledJob(args) {
        try {
            logger.info("starting");
            if (!this.isStarting) {
                this.isStarting = true;

                const runningCount = Executor.tasks.length; //Executor.tasks is a singleton containing some info about tasks. details are irrelevant. the point is it contains the active tasks.
                const tasksLimit = 100;

                if (runningCount < tasksLimit) {
                    for await (const i of Array(tasksLimit - runningCount).keys()) {
                        if (Executor.tasks.length > 20)
                            await global.sleep(5 * 1000);

                        this.startWrapper(args); //calling main task here
                    }
                }
                this.isStarting = false;

                logger.info(`Started: ${Executor.tasks.length - runningCount}`);
            }
        } catch (e) {
            logger.error("Error running scheduled job: " + e.toString());
        }
    }

In this example the problem manifests as following: All work as expected first 3-5 hours, later for each time the scheduled function called:

1. logger.info("here scheduledJobWrapper"); does now show any output.
2. logger.info("starting"); not in the output
3. this.startWrapper does run and the code inside it is being executed.

Despite that the code inside of this.startWrapper is still running, the count of newly created jobs is slowly decreasing. Hardware (RAM/CPU) is not getting any significant load (CPU under 10%, RAM under 20%)

Any clue on possible reason?

nodejs: 12.6.0

Thanks!

UPDATE it seems like that with the usage of setInterval the app is running OK for a longer time (6-24 hours), but after that the problem still starts.

Answer 1

The issue is with setInterval function. It gets slow down with the time. It has wierd behavior too. You can create custom setInterval using setTimeout or use third-party module and give try.

Sample setInterval Implementation.

const intervals = new Map();
function setInterval(fn, time, context, ...args) {
  const id = new Date().getTime() + "" + Math.floor(Math.random() * 10000);
  intervals.set(
    id,
    setTimeout(function next() {
      intervals.set(id, setTimeout(next, time));
      fn.apply(context, args);
    }, time)
  );
  return id;
}
function clearInterval(id) {
  clearTimeout(intervals.get(id));
}

setInterval(console.log, 100, console, "hi");

You can also enhance, by adding delta time loss in next setTimeout. Meaning if time loss, run next setTimeout earlier.

Answer 2

First of all, It will be better to move instance of Main() in listen scope:

app.listen(port, () => {
   new Main().run();
});

I don't know how good idea is to run setInterval function in the backend side. It's better to extract this logic and move it in cron job.

Are we sure that the machine can run 100 tasks? Please count the tasks by order and see when the problem starts. Probably you can not schedule 100 tasks and exists one limit somewhere