Unable to use google-cloud in a GAE app

Question

The following line in my Google App Engine app (webapp.py) fails to import the Google Cloud library:

from google.cloud import storage

With the following error:

ImportError: No module named google.cloud.storage

I did some research and found the following articles to be helpful:

• https://cloud.google.com/appengine/docs/python/tools/using-libraries-python-27#installing_a_library
• https://stackoverflow.com/a/34585485
• https://www.simonmweber.com/2013/06/18/python-protobuf-on-app-engine.html

Using a combination of the techniques suggested by the above articles, I did the following:

1. Create a requirements.txt file:

   google-cloud==0.19.0
   

2. Import this library using pip:

   pip install -t lib -r requirements.txt
   

3. Use the following code in my appengine_config.py file:

   import os
   import sys
   import google
   libDir = os.path.join(os.path.dirname(__file__), "lib")
   google.__path__.append(os.path.join(libDir, "google"))
   sys.path.insert(0, libDir)
   

Can anyone shed light on what I might be missing to get this working? I'm just trying to write a Google App Engine app that can write/read from Google Cloud Storage, and I'd like to test locally before deploying.

Answer 1

It looks like the only thing that is required is to include google-cloud into your project requirements.txt file.

Check if this simple sample works for you (you shouldn't get any imports error). Create below files and run pip install -r requirements.txt -t lib. Nothing more is required on my site to make it work.

app.yaml

application: mysample
runtime: python27
api_version: 1
threadsafe: true

handlers:
  - url: /.*
    script: main.app

main.py

import webapp2
from google.cloud import storage


class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Hello, World!')

app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

appengine_config.py

from google.appengine.ext import vendor
import os

# Third-party libraries are stored in "lib", vendoring will make
# sure that they are importable by the application.
if os.path.isdir(os.path.join(os.getcwd(), 'lib')):
    vendor.add('lib')

requirements.txt

google-cloud

Answer 2

There is an App Engine specific Google Cloud Storage API that ships with the App Engine SDK that you can use to work with Cloud Storage buckets.

import cloudstorage as gcs

Is there a reason you didn't use this built-in library, which requires no configuration to load?

Answer 3

Your appengine_config.py only needs to contain:

from google.appengine.ext import vendor
vendor.add('lib')

All the rest you have posted looks fine to me.

Answer 4

The package namespace seems to have been changed as pointed out in this github issue and hasn't been fully fixed. You could install the older version (pip install gcloud) which uses a different namespace & use this import statement instead:

from gcloud import storage

You should also ensure that you're importing vendor libs in your appengine_config.py as pointed out in dyeray's answer.

The issue seems to have been fixed as at version 0.20.0 of google-cloud. So the import statement in the question should work. Just remember to run pip install --upgrade google-cloud

Answer 5

you should move "import google" line to the position after sys.path.insert

Answer 6

I've run into the same problem and took a long time to sort it out.

Setup 1: If your app.yaml is set up like below:

runtime: python27
api_version: 1
threadsafe: no

handlers:
- url: /.*
  script: main.app

libraries:
- name: jinja2
  version: "2.6"
- name: markupsafe
  version: "0.15

And your main.py is registered your script as an app:

from flask import Flask
app = Flask(__name__)
app.config['DEBUG'] = True

@app.route('/')
def hello():
    """Return a friendly HTTP greeting."""
    return 'Hello World!'

You should be able to run just fine on the google cloud.

Setup 2: How I ran into the ImportModule error -
In my app.yaml file I replaced script=main.app with script=main.app and deleted the

from flask import Flask
app = Flask(__name__)
app.config['DEBUG'] = True

from main.py file. As a result, appengine_config.py isn't getting run for setting the sys path. The solution is that you can either follow the pattern like Setup 1 or add the followings to your main.py file.

from google.appengine.ext import vendor
vendor.add('lib')

Then the app should have the right path for importing the third-party packages. Hope it helps.