1

Are there any best practive in running multiple message queue workers in Shopware 6? We disabled the Admin Worker und configured supervisor.d to run 5 processes of messenger:consume and 1 process of scheduled-task:run . We experienced some weird issue: Enqueue seems to have some 20 minute processing time limit (redeliver_after). After 20 minutes of processing a job seems to be considered stale and any other worker is free to take responsibility for it. We also had infinite loops of message dispatching. The whole collaboration between enqueue, symfony/messenger and Shopware is so complex and Shopware documentation is so parse on that.

Marius
  • 31
  • 5
  • 1
    Yes this seems intended. What kind of long queue jobs do you have? Maybe you can increase that timeout or add some additional locking if it is not okay to execute them twice – Alex Jun 01 '21 at 18:09
  • 1
    What we discovered in the last days: Even with just one Worker jobs running more than 20 minutes cause constraint violations on table message_queue_stats. We have one or two jobs which exceeds this limit. We will try to increase this limit. Futhermore we encounter deadlocks in MySQL on enqueue table. I guess that needs further investigation. – Marius Jun 02 '21 at 19:56
  • @Marius did you find the reason for the MYSQL locks? We have the same issue – Isengo Aug 25 '21 at 06:04
  • @lsengo We switched to Redis for queue persistence. You have two options here: Redis Adapter from Symfony Messenger or Redis Adapter for Enqueue. We tried both and it worked better with Enqueues Redis Adapter. – Marius Aug 26 '21 at 16:12
  • Do you want to post an answer? – Alex Sep 11 '21 at 15:48

1 Answers1

0

Due to nobody else posting an answer, I'll answer this one myself. What we discovered: Even with just one worker jobs running more than 20 minutes cause constraint violations on table message_queue_stats. We have one or two jobs which exceeds this limit. We will try to increase this limit. Futhermore we encounter deadlocks in MySQL on enqueue table. I guess that needs further investigation. Finally we switched to Redis for queue persistence. You have two options here: Redis Adapter from Symfony Messenger or Redis Adapter for Enqueue. We tried both and it worked better with Enqueues Redis Adapter. Keep in mind Shopware advises to use RabbitMQ for having multiple workers.

Marius
  • 31
  • 5