Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-534

Spark 2.4.0, kubernetes client mode. TypeError: object of type 'NoneType' has no len() in authenticate_and_accum_updates of pyspark/accumulators.py

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:
      kubernetes on minikube.
      kubernetes version 1.10.0
      Spark 2.4.0 client mode on kubernetes.
      Apache livy 0.5.0

      Description

      step 1) create dataframe.

      df=spark_session.createDataFrame([{'a':1}])
      

      step 2) do a count or collect

      df.count()

      I get this output.

      // Exception happened during processing of request from ('127.0.0.1', 38690)
      Traceback (most recent call last):
      File "/usr/lib/python3.6/socketserver.py", line 317, in _handle_request_noblock
      self.process_request(request, client_address)
      File "/usr/lib/python3.6/socketserver.py", line 348, in process_request
      self.finish_request(request, client_address)
      File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
      self.RequestHandlerClass(request, client_address, self)
      File "/usr/lib/python3.6/socketserver.py", line 721, in __init__
      self.handle()
      File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 266, in handle
      poll(authenticate_and_accum_updates)
      File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 241, in poll
      if func():
      File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 254, in authenticate_and_accum_updates
      received_token = self.rfile.read(len(auth_token))
      TypeError: object of type 'NoneType' has no len()
      
      

      Repeat the step 2, the error no longer comes.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              saivarunvishal Sai Varun Reddy Daram
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: