Localhost: how to get credentials to connect GAE Python 3 app and Datastore Emulator?

Question

I'd like to use the new Datastore Emulator together with a GAE Flask app on localhost. I want to run it in the Docker environment, but the error I get (DefaultCredentialsError) happens with or without Docker.

My Flask file looks like this (see the whole repository here on GitHub):

main.py:

from flask import Flask
from google.cloud import datastore


app = Flask(__name__)


@app.route("/")
def index():
    return "App Engine with Python 3"


@app.route("/message")
def message():
    # auth
    db = datastore.Client()

    # add object to db
    entity = datastore.Entity(key=db.key("Message"))
    message = {"message": "hello world"}
    entity.update(message)
    db.put(entity)

    # query from db
    obj = db.get(key=db.key("Message", entity.id))

    return "Message for you: {}".format(obj["message"])

The index() handler works fine, but the message() handler throws this error:

[2019-02-03 20:00:46,246] ERROR in app: Exception on /message [GET]
Traceback (most recent call last):
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/main.py", line 16, in message
    db = datastore.Client()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 210, in __init__
    project=project, credentials=credentials, _http=_http
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/client.py", line 223, in __init__
    _ClientProjectMixin.__init__(self, project=project)
INFO     2019-02-03 20:00:46,260 module.py:861] default: "GET /message HTTP/1.1" 500 291
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/client.py", line 175, in __init__
    project = self._determine_default(project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 228, in _determine_default
    return _determine_default_project(project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 75, in _determine_default_project
    project = _base_default_project(project=project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/auth/_default.py", line 306, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

I checked the website in the error log and tried the JSON auth file (GOOGLE_APPLICATION_CREDENTIALS), but the result was that my app then connected with a production Datastore on Google Cloud, instead of the local Datastore Emulator.

Any idea how to resolve this?

Answer 1

I managed to solve this problem by adding env vars directly into the Python code (in this case in main.py) and using the Mock library:

import os

import mock
from flask import Flask, render_template, request
from google.cloud import datastore
import google.auth.credentials


app = Flask(__name__)

if os.getenv('GAE_ENV', '').startswith('standard'):
    # production
    db = datastore.Client()
else:
    # localhost
    os.environ["DATASTORE_DATASET"] = "test"
    os.environ["DATASTORE_EMULATOR_HOST"] = "localhost:8001"
    os.environ["DATASTORE_EMULATOR_HOST_PATH"] = "localhost:8001/datastore"
    os.environ["DATASTORE_HOST"] = "http://localhost:8001"
    os.environ["DATASTORE_PROJECT_ID"] = "test"

    credentials = mock.Mock(spec=google.auth.credentials.Credentials)
    db = datastore.Client(project="test", credentials=credentials)

The Datastore Emulator is then run like this:

gcloud beta emulators datastore start --no-legacy --data-dir=. --project test --host-port "localhost:8001"

Requirements needed:

Flask
google-cloud-datastore
mock
google-auth

GitHub example here: https://github.com/smartninja/gae-2nd-gen-examples/tree/master/simple-app-datastore

Answer 2

The fact that credentials are required indicates you're reaching to the actual Datastore, not to the datastore emulator (which neither needs nor requests credentials).

To reach the emulator the client applications (that support it) need to figure out where the emulator is listening and, for that, you need to set the DATASTORE_EMULATOR_HOST environment variable for them. From Setting environment variables:

After you start the emulator, you need to set environment variables so that your application connects to the emulator instead of the production Datastore mode environment. Set these environment variables on the same machine that you use to run your application.

You need to set the environment variables each time you start the emulator. The environment variables depend on dynamically assigned port numbers that could change when you restart the emulator.

See the rest of that section on details about setting the environment and maybe peek at Is it possible to start two dev_appserver.py connecting to the same google cloud datastore emulator?