How to COPY all entities in a Kind in GAE to another Kind without explicitly calling out each property

Question

How do we use function clone_entity() as described in Copy an entity in Google App Engine datastore in Python without knowing property names at 'compile' time to copy the values to an entity in a different Kind? (since the keys get copied as well so cloning happens in the same Kind so the solution at the above link does not work for this particular purpose!)

Tried the following (and other variations but to no avail)

query = db.GqlQuery("SELECT * FROM OrigKind")
results = query.fetch(10);

for user in results:
    new_entry = models.NewKind()
    new_entry_complete_key = new_entry.put()
    new_entry = clone_entity(user, Key=new_entry_complete_key)
    new_entry.put()

(need to copy all entities from OrigKind to NewKind)

Answer 1

You need a modified version of clone_entity:

There are some pitfalls to the original clone method that are discussed in the answers of the original implementation.

def clone_entity(e, to_klass, **extra_args):
  """Clones an entity, adding or overriding constructor attributes.

  The cloned entity will have exactly the same property values as the original
  entity, except where overridden. By default it will have no parent entity or
  key name, unless supplied.

  Args:
    e: The entity to clone
    extra_args: Keyword arguments to override from the cloned entity and pass
      to the constructor.
  Returns:
    A cloned, possibly modified, copy of entity e.
  """
  klass = e.__class__
  props = dict((k, v.__get__(e, klass)) for k, v in klass.properties().iteritems())
  props.update(extra_args)
  return to_klass(**props)

# Use the clone method
query = db.GqlQuery("SELECT * FROM OrigKind")
results = query.fetch(10);

for user in results:
    new_entry = clone_entity(user, NewKind)
    new_entry.put()

Answer 2

I would like to add a couple of things to Shay's answer:

• Treating the case where to_klass doesn't have properties that e does.
• Adapting the clone_entity method to work with ndb.Model

.

def clone_entity(e, to_klass, **extra_args):
    """Clones an entity, adding or overriding constructor attributes.

    The cloned entity will have exactly the same property values as the original
    entity, except where overridden or missing in to_klass. By default it will have 
    no parent entity or key name, unless supplied.

    Args:
      e: The entity to clone
      to_klass: The target class
      extra_args: Keyword arguments to override from the cloned entity and pass
        to the constructor.
    Returns:
      A cloned, possibly modified, instance of to_klass with the same properties as e.
    """
    klass = e.__class__
    props = dict((k, v.__get__(e, klass))
                 for k, v in klass._properties.iteritems()
                 if type(v) is not ndb.ComputedProperty
    )
    props.update(extra_args)
    allowed_props = to_klass._properties
    for key in props.keys():
        if key not in allowed_props:
            del props[key]
    return to_klass(**props)

Answer 3

It just wrote a utility to copy enties from one appid to another and to zip entities of a kind. This utility makes an exact clone, including keys, NDB repeated properties, serving_urls and blobs referenced in the kind. To make this work I have to know the property types of the entities. I use Python 27 and NDB, but the utility also transfers db.Models.

Here is the code to find all the property types for a kind :

    self.kind = 'Books'                                                    # the entities to copy
    self.model_mods = {'Books' : 'models'}                                 # modules to import the model from for a kind

    module = __import__(self.model_mods[self.kind], globals(), locals(), [self.kind], -1)
    self.model_class = getattr(module, self.kind)

    entity = self.model_class()                                            # ndb or db
    if isinstance(entity, ndb.Model): 
        self.ndb = True
        self.query = self.model_class.query()                              # prepare the query to get all the entities 
        self.makePage = self._pager(self.ndbPager)                         # wrap the ndb pager
    elif isinstance(entity, db.Model): 
        self.ndb = False
        self.query = self.model_class.all()
        self.makePage = self._pager(self.dbPager)                          # wrap the db pager
    else :
        raise ValueError('Failed to classify entities of kind : ' + str(self.kind))
    logging.info('Entities of kind : %s inherits from class : %s.Model' 
                 %(self.kind, self.ndb * 'ndb' + (not self.ndb) * 'db'))

    self.data_types = {}                                                   # create a dict of property data types                                         
    for key in self.model_class._properties :                              # the internals of the model_class object
        property_object = getattr(self.model_class, key.split('.')[0])     # strip, so it works for repeated structured properties
        self.data_types[key] = property_object.__class__.__name__          # get the property type
    logging.debug(self.data_types)

In the above code I wrap a pager (paged transfer using a cursor) for db or NDB to transfer the entities between the GAE appid's.

Based on the properties I can encode and decode the properties to transfer the model. To do this I first create a dict of the entities using NDB : entity.to_dict() or db: entity.to_dict(). And I add the key to the dict. Now I can encode the properties of the entity and pickle the result to transfer the encoded entity:

data = pickle.dumps(entity_dict, 1)
encoded_entity = base64.b64encode(data)