Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29536

PySpark does not work with Python 3.8.0

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.4.7, 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: PySpark
    • Labels:
      None
    • Target Version/s:

      Description

      You open a shell and run arbitrary codes:

        File "/.../3.8/lib/python3.8/runpy.py", line 183, in _run_module_as_main
          mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
        File "/.../3.8/lib/python3.8/runpy.py", line 109, in _get_module_details
          __import__(pkg_name)
        File /.../workspace/forked/spark/python/pyspark/__init__.py", line 51, in <module>
          from pyspark.context import SparkContext
        File "/.../spark/python/pyspark/context.py", line 31, in <module>
          from pyspark import accumulators
        File "/.../python/pyspark/accumulators.py", line 97, in <module>
          from pyspark.serializers import read_int, PickleSerializer
        File "/.../python/pyspark/serializers.py", line 71, in <module>
          from pyspark import cloudpickle
        File "/.../python/pyspark/cloudpickle.py", line 152, in <module>
          _cell_set_template_code = _make_cell_set_template_code()
        File "/.../spark/python/pyspark/cloudpickle.py", line 133, in _make_cell_set_template_code
          return types.CodeType(
      TypeError: an integer is required (got type bytes)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: