Why is PyMongo count_documents is slower than count?

Question

In db['TF'] I have about 60 million records.

I need to get the quantity of the records.

If I run db['TF'].count(), it returns at once.

If I run db['TF'].count_documents({}), that is a such long time before I get the result.

However, the count method will be deprecated.

So, how can I get the quantity quickly when using count_documents? Is there some arguments I missed?

I have read the doc and code, but nothing found.

Thanks a lot!

score 17 · Accepted Answer · edited Sep 11 '19 at 06:20

17

This is not about PyMongo but Mongo itself.

count is a native Mongo function. It doesn't really count all the documents. Whenever you insert or delete a record in Mongo, it caches the total number of records in the collection. Then when you run count, Mongo will return that cached value.

count_documents uses a query object, which means that it has to loop through all the records in order to get the total count. Because you're not passing any parameters, it will have to run over all 60 million records. This is why it is slow.

based on @Stennie comment

You can use estimated_document_count() in PyMongo 3.7+ to return the fast count based on collection metadata. The original count() was deprecated because the behaviour differed (estimated vs actual count) based on whether query criteria was provided. The newer driver API is more intentional about the outcome

edited Sep 11 '19 at 06:20

naught101

18,687
19
90
138

answered Sep 08 '18 at 15:37

Amit Wagner

3,134
3
19
35

so the ```count_documents``` is not suitable for this situation. and I should use ```count``` method or call the native mongo function by myself, right? – Threegirl Sep 09 '18 at 01:42
well you can try use count_documents without any params any see if it works as well – Amit Wagner Sep 09 '18 at 11:29
so please upvote and accept the answer this is what we do here in stackoverflow (: – Amit Wagner Sep 09 '18 at 13:10
3

So why is count() being deprecated according to pymongo? count_documents() takes way to long to run on large datasets. – jschmitter Oct 19 '18 at 19:24
7

@jschmitter You can use [`estimated_data_count()`](http://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.estimated_document_count) in PyMongo 3.7+ to return the fast count based on collection metadata. The original `count()` was deprecated because the behaviour differed (estimated vs actual count) based on whether query criteria was provided. The newer driver API is more intentional about the outcome. – Stennie Dec 27 '18 at 02:51

score 2 · Answer 2 · answered Dec 29 '18 at 06:31

As already mentioned here, the behavior is not specific to PyMongo.

The reason is because the count_documents method in PyMongo performs an aggregation query and does not use any metadata. see collection.py#L1670-L1688

pipeline = [{'$match': filter}]
if 'skip' in kwargs:
    pipeline.append({'$skip': kwargs.pop('skip')})
if 'limit' in kwargs:
    pipeline.append({'$limit': kwargs.pop('limit')})
pipeline.append({'$group': {'_id': None, 'n': {'$sum': 1}}})
cmd = SON([('aggregate', self.__name),
           ('pipeline', pipeline),
           ('cursor', {})])
if "hint" in kwargs and not isinstance(kwargs["hint"], string_type):
    kwargs["hint"] = helpers._index_document(kwargs["hint"])
collation = validate_collation_or_none(kwargs.pop('collation', None))
cmd.update(kwargs)
with self._socket_for_reads(session) as (sock_info, slave_ok):
    result = self._aggregate_one_result(
        sock_info, slave_ok, cmd, collation, session)
if not result:
    return 0
return result['n']

This command has the same behavior as the collection.countDocuments method.

That being said, if you willing to trade accuracy for performance, you can use the estimated_document_count method which on the other hand, send a count command to the database with the same behavior as collection.estimatedDocumentCount See collection.py#L1609-L1614

if 'session' in kwargs:
    raise ConfigurationError(
        'estimated_document_count does not support sessions')
    cmd = SON([('count', self.__name)])
    cmd.update(kwargs)
    return self._count(cmd)

Where self._count is a helper sending the command.

Why is PyMongo count_documents is slower than count?

2 Answers2

Linked