K8s pod memory higher than process requires

Question

I have node.js application that is logging inside memory usage.

rss: 161509376, 
heapTotal: 97697792, 
heapUsed: 88706896, 
external: 733609

And command kubectl top pod which saying how many memory pod is using.

NAME                              CPU(cores)   MEMORY(bytes)
api-596d754fc6-s7xvc              2m           144Mi

As you can see, node app using only 93 MB of memory, while k8s saying that pod consumes 144 MB of memory.

We are using alpine as a base image for the node.js app. I checked the raw alpine image with all dependencies installed without actual application running, and it consumed about 4-8 MB of memory. Deployment has limits set.

...
resources:
  limits:
    memory: 400Mi
    cpu: 2
  requests:
    memory: 90Mi
    cpu: 100m

So, requested memory is lower than one that k8s showing to me. I expect to see, that there would be something closer to actual memory consumption, let's say 100 MB.

How can I understand where this additional memory come from? Why are these numbers having a difference?

All tests have been launched on a single pod (single service has a single pod, no mistakes here).

Update 1.

FROM node:8-alpine

ENV NODE_ENV development
ENV PORT XXXX

RUN echo https://repository.fit.cvut.cz/mirrors/alpine/v3.8/main > /etc/apk/repositories; \
    echo https://repository.fit.cvut.cz/mirrors/alpine/v3.8/community >> /etc/apk/repositories

RUN apk update && \
    apk upgrade && \
    apk --no-cache add git make gcc g++ python

RUN apk --no-cache add vips-dev fftw-dev build-base \
    --repository https://repository.fit.cvut.cz/mirrors/alpine/edge/testing/ \
    --repository https://repository.fit.cvut.cz/mirrors/alpine/edge/main

WORKDIR /app

COPY ./dist /app

RUN npm install --only=production --unsafe-perm

RUN apk del make gcc g++ python build-base && \
    rm /var/cache/apk/*

EXPOSE XXXX

CMD node index.js

Docker image looking like so.

What does your pod spec look like? You have a single container? — Rico, Dec 03 '18 at 18:06

Rico · Answer 1 · 2018-12-03T18:43:03.870

0

So the only other aspect that consumes memory in Node.js (and other language runtimes) is the garbage collector. You didn't describe that whether you see an upwards consumption pattern but if you do, it's possible that you have some sort of leak. If your consumption remains stable it's possible that's the given amount consumed by the garbage collector for your specific application. For more information on when what the garbage collector is doing you can use the node-gc-profiler.

This blog sheds light on the Node.js, memory consumption and garbage collection. There are also a ton of online resources on how to troubleshoot Node.js memory usage and garbage collection. [1], [2], [3], etc.

Hope it helps!

edited Dec 03 '18 at 18:43

answered Dec 03 '18 at 18:37

Rico

58,485
12
111
141

That is the point, I spent whole weekend to figure out where is the leak, and what I'm seeing right now, that profiler showing me even less, then process.memoryUsage() does. I would not say, that GC here working as some standalone process, it is already included in the total process memory. I do not see memory growth (I see short drops, which saying that GC is working) and it stays more or less constantly even if I'm making several requests, which once again saying, that there is no leaks in this module. – Artsiom Miksiuk Dec 03 '18 at 18:55
hmm, have you profiled your app with pmap or valgrind? I'm not sure that it's something that k8s or docker is sticking in there. – Rico Dec 03 '18 at 19:04
no. Don't think that it will help, but I don't have anything left here. – Artsiom Miksiuk Dec 03 '18 at 19:12

score 0 · Answer 2 · answered Sep 24 '19 at 18:32

You are most likely seeing a merge of the pod actual memory consumption and the buffer/cache memory that the kernel use for file caching.

This is probably a bug in Kubernetes itself. See this issue : https://github.com/kubernetes/kubernetes/issues/43916

This will happen if your pod read/write files. If your pod ever reach the limit, the kernel will wipe it's buffer-cache before oomkiller activate so it's not too dangerous to set a hard limit. If you do not set a limit, the node will eventually stop scheduling pods or even restart pods that are consuming too much "memory".

K8s pod memory higher than process requires

2 Answers2