Query random row in ndb

Question

I am trying to develop an web app on google app engine using webapp2.

One of the thing I need to do is to retrieve a random from ndb and display it. Is it any effective method that allows me to do so?

possible duplicate of [Fetching a random record from the Google App Engine Datastore?](http://stackoverflow.com/questions/3002999/fetching-a-random-record-from-the-google-app-engine-datastore) — Sebastian Kreft, Jun 25 '13 at 18:06

Tim Hoffman · Accepted Answer · 2013-06-26T11:02:57.387

3

I assume you mean a random record when you say "random from ndb".

If you are using automatic id's you could use the following approach. (how sparse you id's will affect how successful this will be).

use random.randrange(start, stop) with start being 0, stop being (2^52)-1 , given the new id allocation policy.

do a keys only query for keys greater than key created from random id. if no results try getting keys < key created.

fetch 10 (or some number) of keys

do a random choice random.choice(seq) on the sequence of keys returned from the earlier fetch.

key.get() the chosen record.

The alternative for a small number of entities say < 1000

do a keys only query and fecth all the keys, then do a random.choice() on the list of keys and the a db.get() on the chosen key. This will be much quicker than any looping solution. If you do this a lot and the set of entities to choose from do not change to frequently and the list of keys is less than 1MB in size, you could cache the keys in memcache.

edited Jun 26 '13 at 11:02

answered Jun 25 '13 at 07:09

Tim Hoffman

12,976
1
17
29

hey, following this answer suggestion, I implemented a super simplified version since fetching with a randrange and query for key < or > didn't work for me (either empty or same value). here's a snippet: 1. fetch all keys `keys = Model.query().fetch(keys_only=True)` 2. grab a random key `key = random.sample(keys, 1)[0]` 3. get the entity: `return key.get()`. of course this could be easily extended to support ranges of entities. – gru Jan 30 '14 at 12:56
How well a particular approach will depend heavily on the number of and sparseness of allocated id's. The fact you can fetch all the keys in a single query falls into my small number of entity qualifiers. The other approach will be for millions and more keys. – Tim Hoffman Jan 30 '14 at 14:02

score 3 · Answer 2 · answered Jun 25 '13 at 17:34

3

There's a scatter reserved property. I don't know much about it, but it's mentioned in the map/reduce implementation.

answered Jun 25 '13 at 17:34

Guido van Rossum

16,690
3
46
49

I not too sure what is scatter reserved property – Jun Hao Jun 27 '13 at 09:40
1

Have you tried Googling for it? – Guido van Rossum Jul 04 '13 at 17:04
It's only possible to get the records from ndb ordered by `__scatter__` property. ndb doesn't allow retrieving the actual property AFAIK. So the results are going to be the same always. There's no randomness in it. – yaswanth Aug 27 '19 at 15:10

score 0 · Answer 3 · answered Jun 25 '13 at 10:23

0

If the number of entities in your datastore is not very large, then you can use the below approach: 1, Use yourquery = entitykind.query() to fetch all entities

2, Use yourquery.count() to get the number of entities

3, Use a random number generator to create a random number within the above count value

4, Use a for loop to iterate through the entities returned by

yourquery.fetch()

and then when the number of times the loop has been executed equals the above random number, use the particular entity in your webapp

answered Jun 25 '13 at 10:23

tony m

4,769
1
21
28

This works fine for small sets of data. If you have a large number entities this won't scale. – Tim Hoffman Jun 25 '13 at 12:25
Even for small sets there's a better way, using offset=, limit=1. – Guido van Rossum Jun 25 '13 at 17:33

Query random row in ndb

3 Answers3

Linked