[SPARK-27927] driver pod hangs with pyspark 2.4.3 and master on kubenetes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.3, 3.0.0
• Component/s: Kubernetes, PySpark, Spark Core

Environment

k8s 1.11.9

spark 2.4.3 and master branch.

• Assignee: Unassigned
• Reporter: Edwin Biemond
• Created: 03/Jun/19 10:52
• Updated: 12/Dec/22 18:11

Description

When we run a simple pyspark on spark 2.4.3 or 3.0.0 the driver pods hangs and never calls the shutdown hook. 

#!/usr/bin/env python

from __future__ import print_function

import os
import os.path
import sys

# Are we really in Spark?
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('hello_world').getOrCreate()
print('Our Spark version is {}'.format(spark.version))
print('Spark context information: {} parallelism={} python version={}'.format(
str(spark.sparkContext),
spark.sparkContext.defaultParallelism,
spark.sparkContext.pythonVer
))

When we run this on kubernetes the driver and executer are just hanging. We see the output of this python script. 

bash-4.2# cat stdout.log
Our Spark version is 2.4.3
Spark context information: <SparkContext master=k8s://https://kubernetes.default.svc:443 appName=hello_world> parallelism=2 python version=3.6

What works

• a simple python with a print works fine on 2.4.3 and 3.0.0
• same setup on 2.4.0
• 2.4.3 spark-submit with the above pyspark