[SPARK-33897] Can't set option 'cross' in join method.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: GokiMori
• Reporter: GokiMori
• Created: 24/Dec/20 05:27
• Updated: 12/Dec/22 18:10
• Resolved: 26/Dec/20 07:34

Description

The PySpark documentation says "Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti."
However, I get the following error when I set the cross option.

 

scala> val df1 = spark.createDataFrame(Seq((1,"a"),(2,"b")))
df1: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
scala> val df2 = spark.createDataFrame(Seq((1,"A"),(2,"B"), (3, "C")))
df2: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
scala> df1.join(right = df2, usingColumns = Seq("_1"), joinType = "cross").show()
java.lang.IllegalArgumentException: requirement failed: Unsupported using join type Cross
 at scala.Predef$.require(Predef.scala:281)
 at org.apache.spark.sql.catalyst.plans.UsingJoin.<init>(joinTypes.scala:106)
 at org.apache.spark.sql.Dataset.join(Dataset.scala:1025)
 ... 53 elided

 

Issue Links

• links to: [Github] Pull Request #30803 (kozakana)