redis memory and cpu spikes

Question

We use redis for some data in our app, and it's totally great. I noticed however occasional cpu and memory spikes on the redis-server process.

This is the Giraffe dashboard from both our production and staging environments. Staging is obviously much less busy, but production isn't terribly busy either normally...

This seems to correlate with the background saving, but not with all of them. Only a handful of those create this spike. Perhaps all do, and it's only down to the measurement resolution though (some are simply not caught within our memory/cpu monitoring cycle). I'm not entirely sure.

I'm still wondering whether this is expected / normal. We don't observe any issues, but I want to be on the safe side. If we have more traffic/activity on our production, are we likely to see many more spikes like these?

UPDATE:

redis log file around the time of the spike

[18588] 05 May 11:42:51.004 * 10 changes in 300 seconds. Saving...
[18588] 05 May 11:42:51.258 * Background saving started by pid 32712
[32712] 05 May 11:43:00.511 * DB saved on disk
[32712] 05 May 11:43:00.549 * RDB: 1 MB of memory used by copy-on-write
[18588] 05 May 11:43:00.629 * Background saving terminated with success

Answer 1

From experimenting further with this and reading about redis persistence, I think the following observations can be made:

• When using RDB (the default settings), redis will fork each time a save operation is triggered, which (by default) is set to once every 15 minutes as a minimum. When more writes to Redis are performed, then RDB writes are as frequent as once every 60 seconds.
• Each fork will use a "copy-on-write" memory allocation, which means that whilst memory won't actually double - it will appear so on tools like ps, htop and the like.
• The fork itself can be quite a cpu-intensive operation, particularly on xen-based virtual hosts (which is what we're using currently).
• The write operation seems to completely overwrite the existing RDB file. It does not write only the changes, but rather dumps the entire dataset to disk.

So on a modest virtual host with 4Gb RAM and data set of around 750Mb (at the time I posted the question), this starts to become rather "expensive". We observed those CPU/Memory spikes, as well as increased IO, even under fairly moderate load / redis usage.

So to answer my own question - this does seem to be the "expected" behaviour.

As for improving the situation, we opted to switch our configuration to use a combination of RDB and AOF. AOF (Append Only File), does appear to only write changes to disk. You can (and should) still configure the AOF file to rewrite (using auto-aof-rewrite-percentage and auto-aof-rewrite-min-size settings). It is also advisable to still use RDB for snapshots. In this configuration however, you can probably do full rewrites / snapshots less frequently and still maintain pretty-good performance and even-better durability.

Answer 2

The docs says: "the Redis AOF works incrementally updating an existing state, like MySQL or MongoDB does, while the RDB snapshotting creates everything from scratch again and again, that is conceptually more robust."

Source: http://redis.io/topics/persistence (in AOF disadvantages)

Answer 3

If I recall correctly, redis forks the process when it does a background save, but only duplicates the memory that is being changed while the save is in progress. So the bump in CPU/memory would depend heavily on how much of the data is being changed while the save is running. So it could certainly be a huge spike at times, and a much lesser spike other times (or none at all, depending on how your load looks).