Python asyncio - Loop exits with Task was destroyed but it is pending

Question

This is the relevant code of my python program:

import discord
import asyncio

class Bot(discord.Client):
    def __init__(self):
        super().__init__()

    @asyncio.coroutine
    def my_background_task(self):
        yield from self.wait_until_ready()
        while not self.is_closed:
            yield from asyncio.sleep(3600*24) # <- This is line 76 where it fails
            doSomething()

bot = Bot()
loop = asyncio.get_event_loop()
try:
    loop.create_task(bot.my_background_task())
    loop.run_until_complete(bot.login('username', 'password'))
    loop.run_until_complete(bot.connect())
except Exception:
    loop.run_until_complete(bot.close())
finally:
    loop.close()

The program occasionally quits (on its own, while it should not) with no other errors or warning other than

Task was destroyed but it is pending!
task: <Task pending coro=<my_background_task() running at bin/discordBot.py:76> wait_for=<Future pending cb=[Task._wakeup()]>>

How to ensure the program won't randomly quit? I have Python 3.4.3+ on Xubuntu 15.10.

Answer 1

This is because the discord client module needs control once every minute or so.

This means that any function that steals control for more than a certain time causes discord's client to enter an invalid state (which will manifest itself as an exception some point later, perhaps upon next method call of client).

To ensure that the discord module client can ping the discord server, you should use a true multi-threading solution.

One solution is to offload all heavy processing onto a separate process (a separate thread will not do, because Python has a global interpreter lock) and use the discord bot as a thin layer whose responsibility is to populate work queues.

Related reading: https://discordpy.readthedocs.io/en/latest/faq.html#what-does-blocking-mean

Example solution... this is WAY beyond the scope of the problem, but I already had the code mostly written. If I had more time, I would write a shorter solution :)

2 parts, discord interaction and processing server:

This is the discord listener.

import discord
import re
import asyncio
import traceback

import websockets
import json

# Call a function on other server
async def call(methodName, *args, **kwargs):
    async with websockets.connect('ws://localhost:9001/meow') as websocket:
        payload = json.dumps( {"method":methodName, "args":args, "kwargs": kwargs})
        await websocket.send(payload)
        #...
        resp = await websocket.recv()
        #...
        return resp

client = discord.Client()
tok = open("token.dat").read()

@client.event
async def on_ready():
    print('Logged in as')
    print(client.user.name)
    print(client.user.id)
    print('------')

@client.event
async def on_error(event, *args, **kwargs):
    print("Error?")

@client.event
async def on_message(message):
    try:
        if message.author.id == client.user.id:
            return
        m = re.match("(\w+) for (\d+).*?", message.content)
        if m:
            g = m.groups(1)
            methodName = g[0]
            someNumber = int(g[1])
            response = await call(methodName, someNumber)
            if response:
                await client.send_message(message.channel, response[0:2000])
    except Exception as e:
        print (e)
        print (traceback.format_exc())

client.run(tok)

This is the worker server for processing heavy requests. You can make this part sync or async.

I chose to use some magic called a websocket to send data from one python process to another one. But you can use anything you want. You could make one script write files into a dir, and the other script could read the files out and process them, for example.

import tornado
import tornado.websocket
import tornado.httpserver
import json
import asyncio
import inspect
import time

class Handler:
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def consume(self, text):
        return "You said {0} and I say hiya".format(text)

    async def sweeps(self, len):
        await asyncio.sleep(len)
        return "Slept for {0} seconds asynchronously!".format(len)

    def sleeps(self, len):
        time.sleep(len)
        return "Slept for {0} seconds synchronously!".format(len)


class MyService(Handler, tornado.websocket.WebSocketHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def stop(self):
        Handler.server.stop()

    def open(self):
        print("WebSocket opened")

    def on_message(self, message):
        print (message)
        j = json.loads(message)
        methodName = j["method"]
        args = j.get("args", ())

        method = getattr(self, methodName)
        if inspect.iscoroutinefunction(method):
            loop = asyncio.get_event_loop()
            task = loop.create_task(method(*args))
            task.add_done_callback( lambda res: self.write_message(res.result()))
            future = asyncio.ensure_future(task)

        elif method:
            resp = method(*args)
            self.write_message(resp)

    def on_close(self):
        print("WebSocket closed")

application = tornado.web.Application([
    (r'/meow', MyService),
])

if __name__ == "__main__":
    from tornado.platform.asyncio import AsyncIOMainLoop
    AsyncIOMainLoop().install()

    http_server = tornado.httpserver.HTTPServer(application)
    Handler.server = http_server
    http_server.listen(9001)

    asyncio.get_event_loop().run_forever()

Now, if you run both processes in separate python scripts, and tell your bot "sleep for 100", it will sleep for 100 seconds happily! The asyncio stuff functions as a make-shift work queue, and you can properly separate the listener from the backend processing by running them as separate python scripts.

Now, no matter how long your functions run in the 'server' part, the client part will never be prevented from pinging the discord server.

Image failed to upload, but... anyway, this is how to tell the bot to sleep and reply... note that the sleep is synchronous. https://i.stack.imgur.com/VdEdd.png

Answer 2

I don't think problem happens while asyncio.sleep. Anyway you shouldn't suppress exception you got:

bot = Bot()
loop = asyncio.get_event_loop()
try:
    # ...
except Exception as e:
    loop.run_until_complete(bot.close())
    raise e  # <--- reraise exception you got while execution to see it (or log it here)
finally:
    # ...

Answer 3

You have to manually stop your task on exit:

import discord
import asyncio

class Bot(discord.Client):
    def __init__(self):
        super().__init__()

    @asyncio.coroutine
    def my_background_task(self):
        yield from self.wait_until_ready()
        while not self.is_closed:
            yield from asyncio.sleep(3600*24) # <- This is line 76 where it fails
            doSomething()

bot = Bot()
loop = asyncio.get_event_loop()
try:
    task = loop.create_task(bot.my_background_task())
    loop.run_until_complete(bot.login('username', 'password'))
    loop.run_until_complete(bot.connect())
except Exception:
    loop.run_until_complete(bot.close())
finally:
    task.cancel()
    try:
        loop.run_until_complete(task)
    except Exception:
        pass 
    loop.close()