Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27927

driver pod hangs with pyspark 2.4.3 and master on kubenetes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.4.3, 3.0.0
    • None
    • Kubernetes, PySpark, Spark Core
    • None
    • k8s 1.11.9

      spark 2.4.3 and master branch.

    Description

      When we run a simple pyspark on spark 2.4.3 or 3.0.0 the driver pods hangs and never calls the shutdown hook. 

      #!/usr/bin/env python
      
      from __future__ import print_function
      
      import os
      import os.path
      import sys
      
      # Are we really in Spark?
      from pyspark.sql import SparkSession
      spark = SparkSession.builder.appName('hello_world').getOrCreate()
      print('Our Spark version is {}'.format(spark.version))
      print('Spark context information: {} parallelism={} python version={}'.format(
      str(spark.sparkContext),
      spark.sparkContext.defaultParallelism,
      spark.sparkContext.pythonVer
      ))
      

      When we run this on kubernetes the driver and executer are just hanging. We see the output of this python script. 

      bash-4.2# cat stdout.log
      Our Spark version is 2.4.3
      Spark context information: <SparkContext master=k8s://https://kubernetes.default.svc:443 appName=hello_world> parallelism=2 python version=3.6

      What works

      • a simple python with a print works fine on 2.4.3 and 3.0.0
      • same setup on 2.4.0
      • 2.4.3 spark-submit with the above pyspark

       

       

       

      Attachments

        1. executor_threads.log
          14 kB
          Edwin Biemond
        2. driver_threads.log
          55 kB
          Edwin Biemond

        Activity

          People

            Unassigned Unassigned
            ebiemond Edwin Biemond
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: