[SPARK-22916] shouldn't bias towards build right if user does not specify

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Feng Liu
• Reporter: Feng Liu
• Created: 28/Dec/17 00:48
• Updated: 29/Jan/18 02:03
• Resolved: 29/Dec/17 10:49

Description

This is an issue very similar to SPARK-22489. When there are no broadcast hints, the current spark strategies will prefer to build right, without considering the sizes of the two sides. To reproduce:

import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec

spark.createDataFrame(Seq((1, "4"), (2, "2"))).toDF("key", "value").createTempView("table1")
spark.createDataFrame(Seq((1, "1"), (2, "2"), (3, "3"))).toDF("key", "value").createTempView("table2")

val bl = sql(s"SELECT * FROM table1 t1 JOIN table2 t2 ON t1.key = t2.key").queryExecution.executedPlan

The plan is going to broadcast right side (`t2`), even though it is larger.

Issue Links

• links to: [Github] Pull Request #20099 (liufengdb)
• links to: [Github] Pull Request #20420 (gatorsmile)