1

This used to be possible by downloading with the bulkloader and uploading to the local dev server. However, the bulkloader download has been non-functional for several months now, due to not supporting oauth2.

A few places recommend downloading from a cloud storage backup, and uploading to the local datastore through either bulkloader or by directly parsing the backup. However, neither of these appear functional anymore. The bulkloader method throws:

OperationalError: unable to open database file

And the RecordsReader class, which is used to read the backup files, reaches end of file when trying to read the first record, resulting in no records being read.

Does there exist a current, functional, method for copying the live datastore to the local dev datastore?

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
JustinLovinger
  • 782
  • 7
  • 15

2 Answers2

1

RecordsReader is functioning perfectly on unix. I've tried this https://gist.github.com/jehna/3b258f5287fcc181aacf one day ago and it worked amazing.

You should add to the imports your Kinds implementation and run it in the datastore interactive shell. for example: from myproject.kinds_implementations import MyKind I've removed the

for pp in dir(a): try: ppp = getattr(a, "_" + pp) if isinstance(ppp, db.Key): ppp._Key__reference.set_app(appname) ppp except AttributeError: """ It's okay """

And it worked well. In my case the backup downloaded in multiple directories so I've modified the access to the directories. for some thing like that:

for directory in mypath: full_directory_path = join(mypath, directory) for sub_dir in listdir(directory_full_path): full_sub_dir_path = join(full_directory_path, sub_dir) onlyfiles = [ f for f in listdir(full_sub_dir_path) if isfile(join(mypath,f)) ] for file in onlyfiles:

If you're working on windows you're welcome to follow my question about RecordsReader on windows, hopefully someone will answer there Google datastore backup to local dev_appserver

edit: Working great on windows if you change the file open permissions from 'r' to 'rb'

Community
  • 1
  • 1
Rotem Vil
  • 224
  • 1
  • 8
1

The bulkloader is still functional on Python with OAuth2, albeit with some caveats. In downloading from the live app, there is an issue with refreshing of the OAuth2 token so the total download time is limited to 3600 seconds, or 3600+3600 if you manually use a refresh token with --oauth2_refresh_token.

When uploading to the development server app, OAuth2 will fail with a 401, so it's necessary to edit google.appengine.ext.remote_api.handler and stub out 'CheckIsAdmin' to always return True as a workaround:

def CheckIsAdmin(self):
  return True
  user_is_authorized = False
  ...

I upvoted the above answer however, as it looks like a more robust solution at this point.

Adam
  • 5,697
  • 1
  • 20
  • 52