[SPARK-11870] Pyspark Streaming will swallow exceptions in TransformFunction and TransformFunctionSerializer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.1, 1.5.2
• Fix Version/s: 1.4.2, 1.5.3, 1.6.0
• Component/s: PySpark
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 19/Nov/15 23:11
• Updated: 12/Jan/16 21:43
• Resolved: 12/Jan/16 21:43

Description

TransformFunction and TransformFunctionSerializer don't rethrow the exception, so when any exception happens, it just return None. This will cause some weird NPE and confuse people.

Issue Links

• links to: [Github] Pull Request #9847 (zsxwing)