Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26990

Difference in handling of mixed-case partition column names after SPARK-26188

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      I noticed that the PR for SPARK-26188 changed how mixed-cased partition columns are handled when the user provides a schema.

      Say I have this file structure (note that each instance of `pS` is mixed case):

      bash-3.2$ find partitioned5 -type d
      partitioned5
      partitioned5/pi=2
      partitioned5/pi=2/pS=foo
      partitioned5/pi=2/pS=bar
      partitioned5/pi=1
      partitioned5/pi=1/pS=foo
      partitioned5/pi=1/pS=bar
      bash-3.2$
      

      If I load the file with a user-provided schema in 2.4 (before the PR was committed) or 2.3, I see:

      scala> val df = spark.read.schema("intField int, pi int, ps string").parquet("partitioned5")
      df: org.apache.spark.sql.DataFrame = [intField: int, pi: int ... 1 more field]
      scala> df.printSchema
      root
       |-- intField: integer (nullable = true)
       |-- pi: integer (nullable = true)
       |-- ps: string (nullable = true)
      scala>
      

      However, using 2.4 after the PR was committed. I see:

      scala> val df = spark.read.schema("intField int, pi int, ps string").parquet("partitioned5")
      df: org.apache.spark.sql.DataFrame = [intField: int, pi: int ... 1 more field]
      scala> df.printSchema
      root
       |-- intField: integer (nullable = true)
       |-- pi: integer (nullable = true)
       |-- pS: string (nullable = true)
      scala>
      

      Spark is picking up the mixed-case column name pS from the directory name, not the lower-case ps from my specified schema.

      In all tests, spark.sql.caseSensitive is set to the default (false).

      Not sure is this is an bug, but it is a difference.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Gengliang.Wang Gengliang Wang
                Reporter:
                bersprockets Bruce Robbins
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: