[SPARK-46568] Python data source options should be a case insensitive dictionary

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 02/Jan/24 11:49
• Updated: 05/Jan/24 06:29
• Resolved: 05/Jan/24 06:28

Description

Data source options are stored as a `CaseInsensitiveStringMap` in Scala, however, its behavior is inconsistent in Python:

class MyDataSource(DataSource):
    def __init__(self, options):
        self.api_key = options.get("API_KEY") # <- This is None

spark.read.format(..).option("API_KEY", my_key).load(...)

Currently, options will not have this "API_KEY" as everything is converted to lowercase on the Scala side. This can be confusing to users.

Issue Links

• links to: GitHub Pull Request #44564