Major Description
In SPARK-27612, one correctness issue was reported. When protocol 4 is used to pickle Python objects, we found that unpickled objects were wrong. A temporary fix was proposed by not using highest protocol.
It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4. It is suspect to this issue.
A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of Unpickler object. We use single Unpickler object to unpickle serialized Python bytes. Stored objects intervenes next round of unpickling, if the map is not cleared.
We has two options:
1. Continues to reuse Unpickler, but calls its close after each unpickling.
2. Not to reuse Unpickler and create new Unpickler object in each unpickling.
Note: This issue is because internal object map in Pyrolite is not cleared after op code STOP. If we use protocol 4 to pickle Python objects, op code MEMOIZE will store objects in the map. We need to clear up it to make sure next unpickling works on clear map. For now, we can manually clear the map.
Attachments
Issue Links
- relates to
-
SPARK-27612 Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None
-
- Resolved
-
- links to