Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32431

The .schema() API behaves incorrectly for nested schemas that have column duplicates in case-insensitive mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.6, 3.0.0
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The code below throws org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema: `camelcase`; for multiple file formats due to a duplicate column in the requested schema.

      import org.apache.spark.sql.types._
      spark.conf.set("spark.sql.caseSensitive", "false")
      val formats = Seq("parquet", "orc", "avro", "json")
      val caseInsensitiveSchema = new StructType().add("LowerCase", LongType).add("camelcase", LongType).add("CamelCase", LongType)
      formats.map{ format =>
          val path = s"/tmp/$format"
          spark
          .range(1L)
          .selectExpr("id AS lowercase", "id + 1 AS camelCase")
          .write.mode("overwrite").format(format).save(path) 
          spark.read.schema(caseInsensitiveSchema).format(format).load(path).show
      }
      

      Similar code with nested schema behaves inconsistently across file formats and sometimes returns incorrect results:

      import org.apache.spark.sql.types._
      spark.conf.set("spark.sql.caseSensitive", "false")
      val formats = Seq("parquet", "orc", "avro", "json")
      val caseInsensitiveSchema = new StructType().add("StructColumn", new StructType().add("LowerCase", LongType).add("camelcase", LongType).add("CamelCase", LongType))
      formats.map{ format =>
          val path = s"/tmp/$format"
          spark
          .range(1L)
          .selectExpr("NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS StructColumn")
          .write.mode("overwrite").format(format).save(path)
          
          spark.read.schema(caseInsensitiveSchema).format(format).load(path).show
      }
      

      The desired behavior likely should be returning an exception just like in the flat schema scenario.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Max Gekk
                Reporter:
                mswit Michał Świtakowski
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: