Get Pycharm to see dynamically generated python modules

Question

Using Apache Airflow they install plugins by modifying sys.modules like this:

 sys.modules[operators_module.__name__] = operators_module

Which is how they get python classes from their plugins folder to be imported via

from airflow.operators.plugin_name import Operator

even though the class Operator exists inside

airflow/plugins/Operators.py

This makes it impossible for PyCharm to understand the above import statement because it is a non-traditional way of generating module/module name.

Is there any way to get PyCharm to have something like a module/package alias for situations like these?

Answer 1

If your plugin is in the same repository as the dag you can wrap you import in try-except block:

try:
   # local development
   from plugin_name import Operator
except ImportError:
   from airflow.operators.plugin_name import Operator

my_operator = Operator(...)

And mark plugins directory as sources in your PyCharm Project Structure.
Afterwards PyCharm picks up your plugin source code and works nicely with autocompletion and so on.

Answer 2

Airflow loads plugins by searching the plugins directory for AirflowPlugin subclasses, then binding the operators etc. stored in attributes of those subclasses to new modules under the airflow namespace.

Resolving code transforming classes into modules is beyond PyCharm's capabilities, so unless you want to override this behavior by adding the plugins directory to sys.path and importing the contents directly, the best solution is to whitelist the modules airflow creates through Settings -> Editor -> Inspections -> Python -> Unresolved references and adding airflow.operators.plugin_name to the Ignore references list. Then, using the module like so:

from airflow.operators import plugin_name
task = plugin_name.Operator()

will no longer raise any errors, although you still won't get the benefits of code completion.

Answer 3

So far the best solution I've found (which is not a very good solution) is to simply ignore all unresolved attributes in each DAG file. You can do this if you hover over the error, select "More actions..." and then "Mark all unresolved attributes of [DAG FILE NAME] as ignored." Screenshots below.

This is not a very good solution because it sort of defeats the purpose of an IDE, so I'm hoping eventually somebody who understands the issue better will think of a better solution and post it here :)

Answer 4

Maybe other solutions exist but referencing @stephen answer you that can

simply ignore all unresolved attributes in each DAG file.

but I found that marking the dags directory as sources_root resolves the issue of unresolved attributes in an IDE (I am using specifically IntelliJ and Pycharm but could work for other IDEs as well)