[SPARK-24809] Serializing LongHashedRelation in executor may result in data error

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0, 2.3.0
• Fix Version/s: 2.1.4, 2.2.3, 2.3.2, 2.4.0
• Component/s: SQL
• Labels: correctness

Environment

Spark 2.2.1

hadoop 2.7.1

• Assignee: Lijia Liu
• Reporter: Lijia Liu
• Created: 15/Jul/18 10:48
• Updated: 29/Jul/18 20:16
• Resolved: 29/Jul/18 20:16

Description

When join key is long or int in broadcast join, Spark will use LongHashedRelation as the broadcast value. Details see SPARK-14419. But, if the broadcast value is abnormal big, executor will serialize it to disk. But, data will lost when serializing.

Issue Links

• links to: [Github] Pull Request #21772 (liutang123)