Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-741

Unable to submit python code using Python Client

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: API, Batch
    • Labels:
      None
    • Environment:
      Spark - 2.3.0.2.6.5.1175-1
      hdp - 2.6.5.1175-1
    • Docs Text:
      Hide
      Stack Trace:
          self.run()
        File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\threading.py", line 1177, in run
          self.function(*self.args, **self.kwargs)
        File "c:\users\shreyas.kudale\incubator-livy\python-api\src\main\python\livy\job_handle.py", line 236, in _run
          self.executor.submit(self.polling_job)
        File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\thread.py", line 163, in submit
          raise RuntimeError('cannot schedule new futures after shutdown')
      RuntimeError: cannot schedule new futures after shutdown
      Show
      Stack Trace:     self.run()   File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\threading.py", line 1177, in run     self.function(*self.args, **self.kwargs)   File "c:\users\shreyas.kudale\incubator-livy\python-api\src\main\python\livy\job_handle.py", line 236, in _run     self.executor.submit(self.polling_job)   File "C:\Users\shreyas.kudale\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\thread.py", line 163, in submit     raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown

      Description

      • We want to remotely trigger a pySpark job using rest call via Livy. For this, we tried the Python Http Client but getting various errors due to the python version. Since not much is clear through documentation, we wanted to know whether the Livy Code base would be updated for Python 3 or not since Python 2 has been already deprecated.
      • Http Client setup.py is still showing Python2 as a requirement in the setup.py file and hence not sure which version is to be used.  We tried using both Python versions but it's not working out. 
      • With Python2, we get the error shown in the error1.png file while with Python3, the error shown in error2.png occurs but gets resolved when upgraded to Python 3. The code snippet is also attached to the email.

        Attachments

        1. error2.png
          33 kB
          Shreyas Kudale
        2. error1.PNG
          24 kB
          Shreyas Kudale
        3. CodeSnippet.PNG
          14 kB
          Shreyas Kudale

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shreyaskudale Shreyas Kudale
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: