[SPARK-32832] Use CaseInsensitiveMap for DataStreamReader/Writer options

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.6, 3.0.1
• Fix Version/s: 2.4.8, 3.0.2, 3.1.0
• Component/s: Structured Streaming
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 09/Sep/20 16:28
• Updated: 11/Sep/20 04:07
• Resolved: 10/Sep/20 06:48

Description

When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use CaseInsensitiveMap instead of HashMap to fix this bug fundamentally.

spark.readStream
  .option("paTh", "1")
  .option("PATH", "2")
  .option("Path", "3")
  .option("patH", "4")
  .option("path", "5")
  .load()
...
org.apache.spark.sql.AnalysisException:
Path does not exist: file:/.../1;

Issue Links

• relates to: SPARK-32364 Use CaseInsensitiveMap for DataFrameReader/Writer options
• links to: [Github] Pull Request #29702 (dongjoon-hyun)
• links to: [Github] Pull Request #29702 (dongjoon-hyun)
• links to: [Github] Pull Request #29707 (dongjoon-hyun)
• links to: [Github] Pull Request #29707 (dongjoon-hyun)