Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33897

Can't set option 'cross' in join method.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The PySpark documentation says "Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti."
      However, I get the following error when I set the cross option.

       

      scala> val df1 = spark.createDataFrame(Seq((1,"a"),(2,"b")))
      df1: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
      scala> val df2 = spark.createDataFrame(Seq((1,"A"),(2,"B"), (3, "C")))
      df2: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
      scala> df1.join(right = df2, usingColumns = Seq("_1"), joinType = "cross").show()
      java.lang.IllegalArgumentException: requirement failed: Unsupported using join type Cross
       at scala.Predef$.require(Predef.scala:281)
       at org.apache.spark.sql.catalyst.plans.UsingJoin.<init>(joinTypes.scala:106)
       at org.apache.spark.sql.Dataset.join(Dataset.scala:1025)
       ... 53 elided
      
      

       

        Attachments

          Activity

            People

            • Assignee:
              kozakana GokiMori
              Reporter:
              kozakana GokiMori
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: