[SPARK-35454] Ambiguous self-join doesn't fail after transfroming the dataset to dataframe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1
• Fix Version/s: 3.2.0, 3.1.3
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 20/May/21 03:27
• Updated: 28/May/21 13:05
• Resolved: 21/May/21 09:22

Description

test("SPARK-28344: fail ambiguous self join - Dataset.colRegex as column ref") {
  val df1 = spark.range(3)
  val df2 = df1.filter($"id" > 0)

  withSQLConf(
    SQLConf.FAIL_AMBIGUOUS_SELF_JOIN_ENABLED.key -> "true",
    SQLConf.CROSS_JOINS_ENABLED.key -> "true") {
    assertAmbiguousSelfJoin(df1.join(df2, df1.colRegex("id") > df2.colRegex("id")))
  }
}

For this unit test, if we append `.toDF()` to both df1 and df2, the query won't fail. 

Issue Links

• relates to: SPARK-33536 Incorrect join results when joining twice with the same DF
• links to: [Github] Pull Request #32616 (Ngone51)
• links to: [Github] Pull Request #32692 (Ngone51)