7

Our redis servers are, since yesterday, gradually (200MB/hour) using more memory, while the amount of keys (330K) and their data (132MB redis-rdb-tools) stay about the same.

Output of redis-cli info shows 6.89G used memory?!

redis_version:2.4.10
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:3437
uptime_in_seconds:296453
uptime_in_days:3
lru_clock:1905188
used_cpu_sys:8605.03
used_cpu_user:1480.46
used_cpu_sys_children:1035.93
used_cpu_user_children:3504.93
connected_clients:404
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:7400076728
used_memory_human:6.89G
used_memory_rss:7186984960
used_memory_peak:7427443856
used_memory_peak_human:6.92G
mem_fragmentation_ratio:0.97
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:0
changes_since_last_save:1672
bgsave_in_progress:0
last_save_time:1403172198
bgrewriteaof_in_progress:0
total_connections_received:3616
total_commands_processed:127741023
expired_keys:0
evicted_keys:0
keyspace_hits:18817574
keyspace_misses:8285349
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1619791
vm_enabled:0
role:slave
master_host:***BLOCKED***
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
db0:keys=372995,expires=372995
db6:keys=68399,expires=68399

The problem started when we updated our (.net) client code from BookSleeve 1.1.0.4 to ServiceStack v3.9.71 to prepare for an upgrade to Redis 2.8. But a lot of other stuff was updated to And our session state store (also redis, but with harbour client) does not show the same symptoms.

Where is all that Redis memory going? How can I troubleshoot it's usage?

Edit: I just restarted this instance and memory returned to 350M and is now climbing again. The top 10 largest objects are still the same size, ranging from 100K to 25M for nr 1. The amount of keys has dropped to 270K (330K earlier).

baskabas
  • 333
  • 3
  • 5
  • 15
  • are there downstream replicas? are there unexpected numbers of keys: is it possibly things aren't being told to expire? – Marc Gravell Jun 19 '14 at 10:49
  • What do you mean with downstream replica's? There are 8 servers in total, one master with two master/slaves that in turn have two slaves each. – baskabas Jun 19 '14 at 11:47
  • one potential cause of unusual memory use is a broken slave, i.e. where the server with high memory has a slave that isn't working properly, and work is backing up – Marc Gravell Jun 19 '14 at 11:49
  • I've viewed the keys with redis-rdb-tools and nothing out of the ordinary is showing. It is strange that the sum of all the data stored in the keys is only a fraction of what redis is using. When restarted it resyncs with a master and starts at about 315M. – baskabas Jun 19 '14 at 11:49
  • Okay, how to identify the broken slave? – baskabas Jun 19 '14 at 11:49
  • firstly, we don't know for sure that this is the issue; what server version is this? do the slaves show anything interesting in their info? – Marc Gravell Jun 19 '14 at 11:52
  • The output I provided in the post is from a problematic slave. Redis 2.4.10 as you can see. I don't see anything out of the ordinary... part from the memory used. I just checked all of the logs and no errors. Logs are at verbose level. – baskabas Jun 19 '14 at 12:01
  • @backabas the question was: does the problematic node **itself have a slave**. It could be backing up downstream replication. – Marc Gravell Jun 19 '14 at 12:02
  • @Marc, yes it does. But there are slaves down the line (I just restarted one) that don't, but they still suffer from these symptoms. – baskabas Jun 19 '14 at 12:04

1 Answers1

6

Here are some sources of "hidden" memory consumption in Redis:

  • Marc already mentioned the buffers maintained by the master to feed the slave. If a slave is lagging behind its master (because it runs on a slower box for instance), then some memory will be consumed on the master.

  • when long running commands are detected, Redis logs them in the SLOWLOG area, which takes some memory. You may want to use the SLOWLOG LEN command to check the number of records you have here.

  • communication buffers can also take memory. As far as I remember, with old versions of Redis (and 2.4 is quite old - you should really upgrade), it was unbounded, meaning that if you transfer a big object at a point, the communication buffer associated to this client connection will grow and never shrink. If there are many clients dealing occasionally with large objects, it could be a possible explanation. If you use commands retrieving very large data from Redis (in one shot), it can be an explanation as well. For instance, a simple KEYS * command applied on a Redis server storing millions of keys will consume a significant amount of memory.

You mentioned that you have objects as big as 25 MB. You have 404 client connections, if each of them needs to access such objects at a point in time, it will consume 10 GB of memory.

Didier Spezia
  • 70,911
  • 12
  • 189
  • 154
  • Interesting info. We've upgraded to Redis 2.8. Have to monitor before jumping to conclusions. Other problems (100% CPU) still arise. I'll run the commands you mentioned first thing tomorrow. – baskabas Jun 19 '14 at 22:25
  • SLOWLOG does show 70+ entries and counting. I'm continuing this investigation in another thread mentioned in the answer. Redis 2.8 solved the memory issue. – baskabas Jun 20 '14 at 13:11
  • There seem to be problems between ServiceStack V3 and Redis 2.4. Memory is now stable around 220M with 500M at peak load. We still are experiencing performance issues, but I'm continuing that investigation in another thread: [How to increase Redis performance when 100% CPU? Sharding? Fastest .Net Client?](http://stackoverflow.com/questions/24317200/how-to-increase-redis-performance-when-100-cpu-sharding-fastest-net-client) – baskabas Jun 20 '14 at 14:11