[SPARK-23192] Hint is lost after using cached data

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.1, 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Xiao Li
• Reporter: Xiao Li
• Created: 23/Jan/18 16:48
• Updated: 23/Feb/18 06:33
• Resolved: 23/Jan/18 22:57

Description

The hint of the plan segment is lost, if the plan segment is replaced by the cached data.

      val df1 = spark.createDataFrame(Seq((1, "4"), (2, "2"))).toDF("key", "value")
      val df2 = spark.createDataFrame(Seq((1, "1"), (2, "2"))).toDF("key", "value")
      df2.cache()
      val df3 = df1.join(broadcast(df2), Seq("key"), "inner")

Hint is lost in df3. The physical join algorithm will not respect the hint due to the loss.

Issue Links

• links to: [Github] Pull Request #20365 (gatorsmile)