[SPARK-45713] Support registering Python data sources

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 27/Oct/23 22:05
• Updated: 31/Oct/23 17:35
• Resolved: 31/Oct/23 05:08

Description

Support registering Python data sources.

Users can register a Python data source and later use reference it using its name.

class MyDataSource(DataSource):
    @classmethod
    def name(cls):
        return "my-data-source"

spark.dataSource.register(MyDataSource)

Users can then use the name of the data source as the format (will be supported in SPARK-45639)

spark.read.format("my-data-source").load()

Issue Links

• links to: GitHub Pull Request #43566
• links to: GitHub Pull Request #43598