Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43348

Support Python 3.8 in PyPy3

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0
    • 3.5.0
    • PySpark
    • None

    Description

      
      pypy3 version is: Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13)
      [PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
      Starting test(pypy3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /__w/spark/spark/python/target/f1cacde7-d369-48cf-a8ea-724c42872020/pypy3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__rxih6dqu.log)
      Traceback (most recent call last):
        File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 188, in _run_module_as_main
          mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
        File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 111, in _get_module_details
          __import__(pkg_name)
        File "/__w/spark/spark/python/pyspark/__init__.py", line 59, in <module>
          from pyspark.rdd import RDD, RDDBarrier
        File "/__w/spark/spark/python/pyspark/rdd.py", line 54, in <module>
          from pyspark.java_gateway import local_connect_and_auth
        File "/__w/spark/spark/python/pyspark/java_gateway.py", line 32, in <module>
          from pyspark.serializers import read_int, write_with_length, UTF8Deserializer
        File "/__w/spark/spark/python/pyspark/serializers.py", line 69, in <module>
          from pyspark import cloudpickle
        File "/__w/spark/spark/python/pyspark/cloudpickle/__init__.py", line 1, in <module>
          from pyspark.cloudpickle.cloudpickle import *  # noqa
        File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle.py", line 56, in <module>
          from .compat import pickle
        File "/__w/spark/spark/python/pyspark/cloudpickle/compat.py", line 13, in <module>
          from _pickle import Pickler  # noqa: F401
      ModuleNotFoundError: No module named '_pickle' 

      Attachments

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: