Getting data and filtering it takes longer on the backend than on the frontend

Question

I try to minimise the load of data that I get from the backend (w/ an axios get request), by filtering it down to a certain amount. It is data that is comes from different sensors. I only want from the last 24 hours and not from measured so far. That goes into tens of thousands of measured values (stored in the MongoDB database).

On the server side, I tried the following code:

sensorRoutes.get(
  '/', 
  expressAsyncHandler( async(req, res) => {
    const sensorData = await Sensor.find();
    if(sensorData){
      const length = sensorData.length;
      const timeNow = Date.now();
      const last24h = timeNow - 86400000; 
      for(let i = 0; i < length; i++){
        let array = sensorData[i].values.map(item => [ item.value, item.timestamp ]);
        let oneDayData = array.filter(item=> item[1] >= last24h);
        sensorData[i].values = oneDayData;
      }
      res.send(sensorData);
    }else{
      res.status(404).send({message: 'Sensor not found'});
    }
  })
);

This slows down the whole process considerably. The above is on the front-end side now, before using it anywhere else in the App. Doing it on the front-end does not slow down the process as much as doing this at the backend.

My question to you, how can I do this on the backend, to minimise the computing load (and memory usage!) on the frond-end. Your help is much appreciated.

do the filtering in the query https://stackoverflow.com/questions/13314678/mongodb-only-fetch-documents-created-in-the-last-24hrs — Lawrence Cherone, Apr 01 '23 at 21:00
Thank you! This set me on the right path. The solution I found is here bellow. — Aron, Apr 02 '23 at 08:22

score 0 · Answer 1 · answered Apr 02 '23 at 08:39

Thank you Lawrence Cherone The answer you provided set me on the right track. The thing is that I should perform filter actions with MongoDB instead of doing scripting on the arrived data. The solution here is to use aggregate. I collect the values that are greater then a certain value. In this case I store the time of measurement created in milliseconds in the field named timestamp.

The code posted above becomes:



sensorRoutes.get(
  '/', 
  expressAsyncHandler( async(req, res) => {

    const sensorData = await Sensor.aggregate([
      {
        $addFields: {
          values: {
            $filter: {
              input: "$values",
              as: "value",
              cond: {
                $gte: ["$$value.timestamp", new Date(Date.now() - 24 * 60 * 60 * 1000).getTime()]
              }
            }
          }
        }
      }
    ]);
    if(sensorData){
      res.send(sensorData);
    }else{
      res.status(404).send({message: 'Sensor not found'});
    }
  })
);

Here only those values are added to the request (aggregate) of the values array in the format of values: [{value: 23, timestamp: 1678738119782}, {value: 22, timestamp: 1678738117721}, ... etc] Then the $filter function of MongoDB will take the array values as input where the value and timestamp are objects within this array. The following code checks if the condition, that is if the amount of milliseconds is greater then the amount of milliseconds that fit in 24 h. If so, take it.

Getting data and filtering it takes longer on the backend than on the frontend

1 Answers1