0

We are deploying a Spring Boot application in a docker container on Ubuntu 20.x. Along with it are containers running Postgres, Caddy, and Watchtower, all started with Docker Compose.

These are deployed on Digital Ocean droplets with 1GB of RAM. This is how we spin up Spring Boot:

ENTRYPOINT ["java", "-Xmx896m", "-jar", "build.jar"]

Every now and then, our Spring Boot container restarts without warning or explanation. We are logging memory, disk space, CPU, thread count, all the vitals that we can think of. There is no sign of a problem in the JVM that we have been able to recognize. There is also no indication of a problem in the other containers.

We have begun to suspect either Docker or Linux is forcing the container to restart, but we don't know where to look for a log that identifies the perpetrator or the reason why it happened. This is primarily due to a lack of deep knowledge of Docker or Linux. We know enough, apparently, to be dangerous.

It is likely that this has to do with the sizing of our Docker container with respect to the JVM it contains, but without any explanation for the unexplained shutdowns, it is all conjecture. How can we determine what is actually happening?

greymatter
  • 840
  • 1
  • 10
  • 25

1 Answers1

0

When a container is killed because there is no memory left, they receive an OMMKill and all happens silently, Docker can even restart the container and everything looks normal.

You can get this status from OOMKilled showed with the command:

docker container inspect mycontainer

[
    {
        "Id": "ddc71bc1122576c5a123493900ae9cc545d0c46989c9fe7a7259508e28e8b59e",
        "Created": "2021-03-19T04:30:09.168786223Z",
        "Path": "/bin/sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
...

Be careful because Digital Ocean machines doesn't have swap enabled.

Hector Vido
  • 765
  • 5
  • 12
  • Thanks, Hector. Isn't it too late to see OOMKilled once the container has restarted? Is there no log that will show when it was done? – greymatter Mar 21 '21 at 18:18
  • Would you recommend that we always enable swap? – greymatter Mar 21 '21 at 18:19
  • About swap, I don't know, if your application can use it, why not? Its better than a crash. I don't know if status persists between restarts. But you can configure the container to not restart if you done that explicit with `--restart=always`. But I guess that JVM is restarting, not the container for itself. Maybe this can help https://dzone.com/articles/why-my-java-application-is-oomkilled – Hector Vido Mar 21 '21 at 18:25
  • @HectorVido Can you please take a look at the following question: https://stackoverflow.com/q/66849235/9409877 – HKS Mar 29 '21 at 10:09