Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32364

Use CaseInsensitiveMap for DataFrameReader/Writer options

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.6, 3.0.0
    • Fix Version/s: 2.4.7, 3.0.1, 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use CaseInsensitiveMap instead of HashMap to fix this bug fundamentally.

      spark.read
        .option("paTh", "1")
        .option("PATH", "2")
        .option("Path", "3")
        .option("patH", "4")
        .load("5")
      ...
      org.apache.spark.sql.AnalysisException:
      Path does not exist: file:/.../1;
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dongjoon Dongjoon Hyun
                Reporter:
                girish_pandit Girish A Pandit
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: