Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7891

Python class in __main__ may trigger AssertionError

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 1.2.0
    • 2.0.0
    • PySpark
    • None

    • Linux, Python 2.7.3
      pickled by Python pickle Lib

    Description

      Callback functions for spark transformations and actions will be pickled.
      If the callback is instancemethod of _main_ module's class, and the class has more than one instancemethod which using class properties or classmethods, the class will be pickled twice, and 'pickle.memoize' twice, then trigger AssertionError.

      Demo code and traceback attached.

      Attachments

        1. demo_error.log
          5 kB
          flykobe cheng
        2. demo_pickle_error.py
          0.7 kB
          flykobe cheng

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-13697 TransformFunctionSerializer.loads doesn't restore the function's module name if it's '__main__'

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              zsxwing Shixiong Zhu
              flykobe flykobe cheng
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: