Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33897

Can't set option 'cross' in join method.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.1
    • 3.1.0
    • SQL
    • None

    Description

      The PySpark documentation says "Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti."
      However, I get the following error when I set the cross option.

       

      scala> val df1 = spark.createDataFrame(Seq((1,"a"),(2,"b")))
      df1: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
      scala> val df2 = spark.createDataFrame(Seq((1,"A"),(2,"B"), (3, "C")))
      df2: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
      scala> df1.join(right = df2, usingColumns = Seq("_1"), joinType = "cross").show()
      java.lang.IllegalArgumentException: requirement failed: Unsupported using join type Cross
       at scala.Predef$.require(Predef.scala:281)
       at org.apache.spark.sql.catalyst.plans.UsingJoin.<init>(joinTypes.scala:106)
       at org.apache.spark.sql.Dataset.join(Dataset.scala:1025)
       ... 53 elided
      
      

       

      Attachments

        Activity

          People

            kozakana GokiMori
            kozakana GokiMori
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: