[SPARK-3554] handle large dataset in closure of PySpark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Target Version/s: 1.2.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 16/Sep/14 20:45
• Updated: 19/Sep/14 01:12
• Resolved: 19/Sep/14 01:12

Description

Sometimes there are large dataset used in closure and user forget to use broadcast for it, then the serialized command will become huge.

py4j can not handle large objects efficiently, we should compress the serialized command and user broadcast for it if it's huge.

Issue Links

• links to: [Github] Pull Request #2417 (davies)