NodeJs runs out of memory on external data request

Question

I'm doing a large datafetch on my backend with Axios:

await Axios({
      method: 'GET',
      url,
      headers: {
        'content-type': 'application/json',
        Authorization: `Bearer ${forgeAccessToken}`,
      },
    });

NodeJS crashes. Memory problem:

  1|rest-backenq  | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I increased memory from 4GB to 16GB.

No effect.

I've no control of the external API I fetch data from.

What can I do to fix this?

Can you post your server side code, I don't thinks this code may help in this case — Janith, May 15 '19 at 09:13
Don't know since I cant recieve it. But it's > 20mb. It's json-data so it's no file. — Joe, May 15 '19 at 09:20

score 1 · Answer 1 · answered May 15 '19 at 09:22

I think your file is downloading to your memory, use request module instead of Axios to download the file

var fs = require('fs');
var request = require('request');
var progress = require('request-progress');

// The options argument is optional so you can omit it 
progress(request({
    url : url,
    headers: {
        'content-type': 'application/json',
        'Authorization': `Bearer ${forgeAccessToken}`
    }
}), {
    // throttle: 2000,                    // Throttle the progress event to 2000ms, defaults to 1000ms 
    // delay: 1000,                       // Only start to emit after 1000ms delay, defaults to 0ms 
    // lengthHeader: 'x-transfer-length'  // Length header to use, defaults to content-length 
})
.on('progress', function (state) {
    // The state is an object that looks like this: 
    // { 
    //     percent: 0.5,               // Overall percent (between 0 to 1) 
    //     speed: 554732,              // The download speed in bytes/sec 
    //     size: { 
    //         total: 90044871,        // The total payload size in bytes 
    //         transferred: 27610959   // The transferred payload size in bytes 
    //     }, 
    //     time: { 
    //         elapsed: 36.235,        // The total elapsed seconds since the start (3 decimals) 
    //         remaining: 81.403       // The remaining seconds to finish (3 decimals) 
    //     } 
    // } 
    console.log('progress', state);
})
.on('error', function (err) {
    // Do something with err 
})
.on('end', function () {
    // Do something after request finishes 
})
.pipe(fs.createWriteStream('IE11.Win8.1.For.Windows.VirtualBox.zip'));

For more information follow this question, What is way to download big file in NodeJS?

This code is a bit to complicated. I prefer Axios. I incresead node js memory limit and it solved the problem. — Joe, May 15 '19 at 09:41

NodeJs runs out of memory on external data request

1 Answers1