Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27629

Prevent Unpickler from intervening each unpickling

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      In SPARK-27612, one correctness issue was reported. When protocol 4 is used to pickle Python objects, we found that unpickled objects were wrong. A temporary fix was proposed by not using highest protocol.

      It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4. It is suspect to this issue.

      A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of Unpickler object. We use single Unpickler object to unpickle serialized Python bytes. Stored objects intervenes next round of unpickling, if the map is not cleared.

      We has two options:

      1. Continues to reuse Unpickler, but calls its close after each unpickling.
      2. Not to reuse Unpickler and create new Unpickler object in each unpickling.

      Note: This issue is because internal object map in Pyrolite is not cleared after op code STOP. If we use protocol 4 to pickle Python objects, op code MEMOIZE will store objects in the map. We need to clear up it to make sure next unpickling works on clear map. For now, we can manually clear the map.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                viirya L. C. Hsieh
                Reporter:
                viirya L. C. Hsieh
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: