8

I have hard times to demonstrate that the consumer_timeout setting is working as expected. I may have done things wrong or misunderstood the consumer_timeout behavior.

All my code for testing is available here : https://github.com/Rafarel/rabbitmq-tests

Basically, I have a consumer_timeout set to 10000ms (10sec) and then I try to consume the message with a call back that sleeps a bit longer than the timeout value (20sec) before trying to acknowledge the message.

I am supposed to have a PRECONDITION_FAILED exception due to the timeout, but it is not the case. I have the exception if I set the SLEEP_DURATION in receive_timeout.py way more than the consumer_timeout value like 60 seconds.

Quote from https://www.rabbitmq.com/consumers.html#acknowledgement-timeout

If a consumer does not ack its delivery for more than the timeout value (30 minutes by default), its channel will be closed with a PRECONDITION_FAILED channel exception.

If someone could help me understand what I'm doing wrong that would be great, thanks!

Raphaël Roux
  • 316
  • 2
  • 7
  • 1
    https://github.com/Rafarel/rabbitmq-tests/issues/1 – Luke Bakken Feb 02 '22 at 15:37
  • 1
    It looks like the RabbitMQ process that checks for consumer_timeout only runs once per minute. If you increase SLEEP_DURATION to 65 seconds, your channel should be closed. This configuration makes sense to me. It's intended to protect against 'long running' processes. – Chad Knutson Feb 02 '22 at 21:30

2 Answers2

17

Some useful tips:

  1. Dynamic configuration

You can dynamically set the consumer_timeout value by running the following command on the RabbitMQ server:

rabbitmqctl eval 'application:set_env(rabbit, consumer_timeout, 36000000).'

This will set the new timeout to 10 hrs (36000000ms). For this to take effect, you need to restart your workers though. Existing worker connections will continue to use the old timeout.

You can check the current configured timeout value as well:

rabbitmqctl eval 'application:get_env(rabbit, consumer_timeout).'

  1. With Docker images

If you are running RabbitMQ via Docker image, here's how to set the value: Simply add -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit consumer_timeout 36000000" to your docker run OR set the environment RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS to "-rabbit consumer_timeout 36000000".

Sarang
  • 2,143
  • 24
  • 21
  • How can I confirm if this works? I don't see any documentation about this – JSON_Derulo Nov 01 '22 at 09:16
  • Which solution are you referring to: (1) or (2)? In any case, both solutions I have used in production and work fine. Both solutions are derived from official documentation. For example `consumer_timeout` is the name of the variable as per document and then `set_env` and `get_env` are the official methods to set and get value of variables. Let me know if you have any further question. – Sarang Nov 18 '22 at 13:21
  • Thanks so much for the response, I was referring to the second solution. I have gone ahead and implemented it and it seems to be working thus far. But I haven't tested with tasks that have an eta of more than 15 minutes yet. – JSON_Derulo Nov 18 '22 at 14:20
  • 1
    Great! To check if the timeout is set properly, simply run `rabbitmqctl eval 'application:get_env(rabbit, consumer_timeout).'` in your running container. – Sarang Nov 21 '22 at 19:17
  • You could also run `rabbitmqctl environment` to see full configuration including consumer_timeout. – JanDotNet Jun 17 '23 at 07:32
9

For future readers:

The consumer_timeout was never meant to provide any kind of precision, it is there to protect quorum queues mostly and very long running consumers

timeouts will only be evaluated every 60 seconds by default. This interval is controlled by the channel_tick_interval setting (edited)

so try lowering the tick interval to get a bit more precision.

Also your code is blocking the IO: https://github.com/Rafarel/rabbitmq-tests/issues/1

Also

Gabriele Santomaggio
  • 21,656
  • 4
  • 52
  • 52
  • Thanks Gabriele for your answer. I can't find any documentation about this channel_tick_interval so I don't know how to change this setting value? I do not need a very low tick in production, it's just for testing purpose. Thanks! – Raphaël Roux Feb 04 '22 at 10:23
  • 2
    Many thanks. You saved me another hour of detective work :-D For future readers, both `consumer_timeout` and `channel_tick_interval` can be set in advanced config: https://www.rabbitmq.com/configure.html#advanced-config-file, e.g. [ {rabbit, [ {consumer_timeout, 2000}, {channel_tick_interval, 1000} ]} ]. – jenda Feb 23 '22 at 17:05