Start datastore emulator and loading the older locally created datastore.db file

Question

In the process of moving from python 2.7 to python 3 on GAE, we want to start moving the code with minimal risk. One approach that sounds good to us is to migrate our app pages and features in increments running along the Python 2.7 version until it is fully migrated to Python 3.

One challenge that we were facing was running two dev_appserver.py one for our usual Python 2.7, and the other for the new Python 3.11. After some reading (here, dev_appserver docs,..etc.) we were able to start the two using venv on different shells on MacOS.

Then it was the issue of having database locked when running both, since it turned out that both were using the same datastore path. So it seems we need to use DataStore Emulator (docs here) rather than the direct emulator that runs with dev_appserver.py

We had some data in the python 2.7 local development environment that we use for testing, which we used to point at when starting dev_appserver.py by adding the argument

dev_appserver.py app.yaml --datastore_path=<PATH TO datastore.db>

Now that the datastore emulator is running, we want to point it to the old DB file so we can run both dev_appserver.py for Python 2 & python 3 on the same local DB file. This would ensure data consistency of the two running services (Py2 & Py3). We tried specifying the path in the datastore emulator:

gcloud beta emulators datastore start --data-dir=<PATH TO SAME datastore.db>

But it seems to be using a different file format .bin and path PATH/WEB-INF/appengine-generated/local_db.bin so it is not looking for .db files

Our inquiries:

1. How can we load our local datastore.db data in the new running datastore emulator?
2. Will we still have GUI access to the datastore using the datastore emulator like we used to have when running dev_appserver.py through the local url http://localhost:8000/datastore as some older articles mentioned this is still a feature request at the time?
3. Has anyone had difficulty using this approach in the migration to Python3? One challenge we might face is ensuring the consistency of the logged in user session accross different pages/features during running both services at the same time.

There are still many things that we need to figure out along the way of having both versions running along side, but let's take it one step at a time.

Thank you

Answer 1

1. If you launch Python3 with dev_appserver.py, you'll have the GUI which gives you access to datastore, memcache, etc.

2. Emulator stores data as Java Objects while the older one is stored as sqlite3. This is why you have the different formats you referred to(see Google docs ). The same document also says

When dev_appserver is launched with legacy sqlite3 data, the data will be converted to Java objects. The original data is backed up with the filename {original-data-filename}.sqlitestub.

If this isn't happening, then it might be a bug (or some setting is missing). I'll test later to see if I can reproduce the issue/if I have a solution

Update - Tried and the approach below didn't work

An alternative to try

a) Create a copy of your .db file and put in a specific location (different from that of your Python 2)

b) Instead of starting data emulator, just start your Python 3 with dev_appserver normally but with --datastore_path=<PATH TO copy of .db>