[SPARK-28355] Use Spark conf for threshold at which UDF is compressed by broadcast

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Jesse Cai
• Reporter: Jesse Cai
• Created: 11/Jul/19 22:19
• Updated: 14/Jul/19 05:55
• Resolved: 13/Jul/19 15:47

Description

The _prepare_for_python_RDD method currently broadcasts a pickled command if its length is greater than the hardcoded value 1 << 20 (1M). We would like to set this value as a Spark conf instead.

Issue Links

• links to: GitHub Pull Request #25123