3

I'm already familiar with python but I just started to learn about the topic of "asynchronous programming" and "io tasks". In the tutorial, I saw that asyncio.Queue() was used with something that related to producers and consumers, but I didn't understand it that much.

Can someone simplify it, and also give a few examples?

  • 2
    Does this answer your question? [Using asyncio.Queue for producer-consumer flow](https://stackoverflow.com/questions/52582685/using-asyncio-queue-for-producer-consumer-flow) – bad_coder Dec 24 '19 at 07:30
  • Welcome to StackOverflow! Please note that you should ask a question about a concrete problem that you are having rather than ask for help with a class _in general_. As formulated, your question is very hard to answer because it is unclear what your knowledge level and prior experience is. – user4815162342 Dec 29 '19 at 16:18

1 Answers1

6

asyncio.Queues is a rendition of the normal Python queue, but with special asynchronous properties. The following functions are worth-wild paying attention to:

coroutine get()

Remove and return an item from the queue. If queue is empty, wait until an item is available.

coroutine put(item)

Put an item into the queue. If the queue is full, wait until a free slot is available before adding the item.

coroutine join()

Block until all items in the queue have been received and processed.

If you understand the general idea of asynchronous code (code runs in the same thread, but is constantly switching between operations) you should see the general purpose of the asynchronous queue -- which is to allow flexible await operations with get() and put() methods.

Take the example of gathering, filtering, and using some objects (whatever you want them to be). Let us say you asynchronously gather 30 objects where each gathering function will take between 1-5 seconds. Let us also say you want to filter these objects, and filtering takes between 4 to 7 seconds. The filtering function kicks in instantly after the gathering is done, and so perhaps at some point in the asyncio.loop you might have the following operations:

object gathering...
object gathered -> filtering -> stored!
object gathering...
object gathering...
object gathered -> filtering...
object gathered -> filtering...
object gathering...
object gathered -> filtering -> stored!
...

Where the objects denoted with -> filtering are already in the next stage of their operation, and -> stored! have already been gathered and filtered. The usefulness of the asyncio.queue in the scenario above would come when you realize that perhaps it is best to stop filtering once you've reached n amount of stored objects, since filtering hogs the thread and causes you to not be able to effectively use the already filtered objects. Therefore, your stack might end up looking like so:

object gathering -> (waiting for queue to have space)
object gathered -> filtering -> stored!
object gathered -> (waiting for queue to have space)
object gathered -> (waiting for queue to have space)
object gathered -> filtered -> stored!
object gathered -> filtered -> stored!
object gathered -> (waiting for queue to have space)
object gathered -> filtering -> stored!
...

This is one example of millions on how asyncio.queue can be used, but the idea of the example above is for you to start seeing how asynchronous programming is a different dimension of programming in general -- you ought to understand the relevancy of things in the micro and macro scale. Both how the function runs by itself, and how the function interacts with others (in the case of asynchronous function, by not allowing another function to run). Asyncio primatives are the core of this concept, and are worth-wild to be understood.

Community
  • 1
  • 1
felipe
  • 7,324
  • 2
  • 28
  • 37