[SPARK-13697] TransformFunctionSerializer.loads doesn't restore the function's module name if it's '__main__'

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.4.2, 1.5.3, 1.6.2, 2.0.0
• Component/s: PySpark
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 05/Mar/16 08:17
• Updated: 23/Jun/16 21:14
• Resolved: 06/Mar/16 16:58

Description

Here is a reproducer

>>> from pyspark.streaming import StreamingContext
>>> from pyspark.streaming.util import TransformFunction
>>> ssc = StreamingContext(sc, 1)
>>> func = TransformFunction(sc, lambda x: x, sc.serializer)
>>> func.rdd_wrapper(lambda x: x)
TransformFunction(<function <lambda> at 0x106ac8b18>)
>>> bytes = bytearray(ssc._transformerSerializer.serializer.dumps((func.func, func.rdd_wrap_func, func.deserializers))) 
>>> func2 = ssc._transformerSerializer.loads(bytes)
>>> print(func2.func.__module__)
None
>>> print(func2.rdd_wrap_func.__module__)
None
>>> 

Issue Links

• duplicates: SPARK-7891 Python class in __main__ may trigger AssertionError
• links to: [Github] Pull Request #11535 (zsxwing)