PyMongo find query returns empty/partial cursor when running in a Django+uWsgi project

Question

We developed a REST API using Django & mongoDB (PyMongo driver). The problem is that, on some requests to the API endpoints, PyMongo cursor returns a partial response which contains less documents than it should (but it’s a completely valid JSON document).

Let me explain it with an example of one of our views:

def get_data(key):
    return collection.find({'key': key}, limit=24)

def my_view(request):
    key = request.POST.get('key')
    query = get_data(key)
    res = [app for app in query]
    return JsonResponse({'list': res})

We're sure that there is more than 8000 documents matching the query, but in some calls we get less than 24 results (even zero). The first problem we've investigated was that we had more than one MongoClient definition in our code. By resolving this, the number of occurrences of the problem decreased, but we still had it in a lot of calls.

After all of these investigations, we've designed a test in which we made 16 asynchronous requests at the same time to the server. With this approach, we could reproduce the problem. On each of these 16 requests, 6-8 of them had partial results. After running this test we reduced uWsgi’s number of processes to 6 and restarted the server. All results were good but after applying another heavy load on the server, the problem began again. At this point, we restarted uwsgi service and again everything was OK. With this last experiment we have a clue now that when the uwsgi service starts running, everything is working correctly but after a period of time and heavy load, the server begins to return partial or empty results again. The latest investigation we had was to run the API using python manage.py with DEBUG=False, and we had the problem again after a period of time in this situation.

We can't figure out what the problem is and how to solve it. One reason that we can think of is that Django closes pymongo’s connections before completion. Because the returned result is a valid JSON.

Our stack is:

• nginx (with no cache enabled)
• uWsgi
• MemCached (disabled during debugging procedure)
• Django (v1.8 on python 3)
• PyMongo (v3.0.3)

Your help is really appreciated.

Update:

Mongo version:

db version v3.0.7
git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd

• We are running single mongod instance. No sharding/replicating.

• We are creating connection using this snippet:

  con = MongoClient('localhost', 27017)

Update 2

Subject thread in Pymongo issue tracker.

Answer 1

Pymongo cursors are not thread safe elements. So using them like what I did in a multi-threaded environment will cause what I've described in question. On the other hand Python's list operations are mostly thread safe, and changing snippet like this will solve the problem:

def get_data(key):
    return list(collection.find({'key': key}, limit=24))

def my_view(request):
    key = request.POST.get('key')
    query = get_data(key)
    res = [app for app in query]
    return JsonResponse({'list': res})

Answer 2

My very speculative guess is that you are reusing a cursor somewhere in your code. Make sure you are initializing your collection within the view stack itself, and not outside of it.

For example, as written, if you are doing something like:

import ...
import con

collection = con.documents
# blah blah code
def my_view(request):
    key = request.POST.get('key')
    query = collection.find({'key': key}, limit=24)
    res = [app for app in query]
    return JsonResponse({'list': res})

You could end us reusing a cursor. Better to do something like

import ...
import con

# blah blah code
def my_view(request):
    collection = con.documents
    key = request.POST.get('key')
    query = collection.find({'key': key}, limit=24)
    res = [app for app in query]
    return JsonResponse({'list': res})

EDIT at asker's request for clarification:

The reason you need to define the collection within the view stack and not when the file loads is that the collection variable has a cursor, which is basically how the database and your application talk to each other. Cursors do things like keep track of where you are in a long list of data, in addition to a bunch of other stuff, but thats the important part.

When you create the collection cursor outside the view method, it will re-use the cursor on each request if it exists. So, if you make one request, and then another really, really fast right after that (like what happened when you applied high load), the cursor might only be half way through talking to the database, and so some of your data goes to the first request, and some to the second. The reason you would get NO data in a request would be if a cursor finished fetching data but hadn't been closed yet, so the next request tried to fetch data from the cursor, and there was none left to fetch in the query.

By moving the collection definition (and by association, the cursor definition) into the view stack, you will ALWAYS get a new cursor when you process a new request. You wont get any cross talking between your cursors and different requests, as each request cycle will have its own.