[SPARK-32364] Use CaseInsensitiveMap for DataFrameReader/Writer options

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.6, 3.0.0
• Fix Version/s: 2.4.7, 3.0.1, 3.1.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Girish A Pandit
• Created: 20/Jul/20 05:57
• Updated: 09/Sep/20 21:02
• Resolved: 22/Jul/20 15:00

Description

When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use CaseInsensitiveMap instead of HashMap to fix this bug fundamentally.

spark.read
  .option("paTh", "1")
  .option("PATH", "2")
  .option("Path", "3")
  .option("patH", "4")
  .load("5")
...
org.apache.spark.sql.AnalysisException:
Path does not exist: file:/.../1;

Issue Links

• causes: SPARK-32756 Fix CaseInsensitiveMap in Scala 2.13
• is related to: SPARK-32832 Use CaseInsensitiveMap for DataStreamReader/Writer options
• links to: [Github] Pull Request #29160 (dongjoon-hyun)
• links to: [Github] Pull Request #29191 (dongjoon-hyun)
• links to: [Github] Pull Request #29191 (dongjoon-hyun)
• links to: [Github] Pull Request #29209 (dongjoon-hyun)
• links to: [Github] Pull Request #29584 (karolchmist)