9

I am able to run pyspark and run a script on Jupyter notebook. But when I try to run the file from terminal using spark-submit, getting this error:

Error executing Jupyter command file path [Errno 2] No such file or directory

Can anyone help me what's wrong with my config?

I'm using Python 2.7 and Spark 1.6

Q Bit
  • 91
  • 1
  • 4

1 Answers1

29

These problem occurs because you have set jupyter to run pyspark script . Now you should unset PYSPARK_DRIVER_PYTHON variable command for the same is $ unset PYSPARK_DRIVER_PYTHON then try spark-submit again ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999.

Shubham Sharma
  • 2,763
  • 5
  • 31
  • 46
  • I was using the following in spark-env.sh export PYSPARK_PYTHON=python3.7 # export PYSPARK_DRIVER_PYTHON=jupyter # export PYSPARK_DRIVER_PYTHON_OPTS="notebook" If I want to run Jupyter will have to set the driver and run separately pyspark command. Thanks not spark-submit command. – Doogle Dec 02 '18 at 09:16