[LIVY-741] Unable to submit python code using Python Client - ASF JIRA

XML

Word

Printable

JSON

Docs Text:

Hide
Stack Trace:
    self.run()
  File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\threading.py", line 1177, in run
    self.function(*self.args, **self.kwargs)
  File "c:\users\shreyas.kudale\incubator-livy\python-api\src\main\python\livy\job_handle.py", line 236, in _run
    self.executor.submit(self.polling_job)
  File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\thread.py", line 163, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

Show
Stack Trace:     self.run()   File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\threading.py", line 1177, in run     self.function(*self.args, **self.kwargs)   File "c:\users\shreyas.kudale\incubator-livy\python-api\src\main\python\livy\job_handle.py", line 236, in _run     self.executor.submit(self.polling_job)   File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\thread.py", line 163, in submit     raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown

We want to remotely trigger a pySpark job using rest call via Livy. For this, we tried the Python Http Client but getting various errors due to the python version. Since not much is clear through documentation, we wanted to know whether the Livy Code base would be updated for Python 3 or not since Python 2 has been already deprecated.
Http Client setup.py is still showing Python2 as a requirement in the setup.py file and hence not sure which version is to be used. We tried using both Python versions but it's not working out.
With Python2, we get the error shown in the error1.png file while with Python3, the error shown in error2.png occurs but gets resolved when upgraded to Python 3. The code snippet is also attached to the email.