[SPARK-29536] PySpark does not work with Python 3.8.0

• Type: Test
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark
• Target Version/s: 3.0.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 21/Oct/19 12:38
• Updated: 12/Dec/22 18:11
• Resolved: 22/Oct/19 07:18

Description

You open a shell and run arbitrary codes:

  File "/.../3.8/lib/python3.8/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/.../3.8/lib/python3.8/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File /.../workspace/forked/spark/python/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/.../spark/python/pyspark/context.py", line 31, in <module>
    from pyspark import accumulators
  File "/.../python/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/.../python/pyspark/serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "/.../python/pyspark/cloudpickle.py", line 152, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/.../spark/python/pyspark/cloudpickle.py", line 133, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)

Issue Links

• is related to: SPARK-33153 HiveExternalCatalogVersionsSuite fails on Ubuntu 20.04
• links to: GitHub Pull Request #26194