[SPARK-45639] Support loading Python data sources in DataFrameReader

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 23/Oct/23 20:53
• Updated: 13/Nov/23 11:37
• Resolved: 08/Nov/23 17:23

Description

Allow users to read from a Python data source using `spark.read.format(...).load()` in PySpark. For example

Users can extend the DataSource and the DataSourceReader classes to create their own Python data source reader and use them in PySpark:

class MyReader(DataSourceReader):
    def read(self, partition):
        yield (0, 1)

class MyDataSource(DataSource):
    def schema(self):
        return "id INT, value INT"
    
    def reader(self, schema):
        return MyReader()

df = spark.read.format("MyDataSource").load()
df.show()
+---+-----+
| id|value|
+---+-----+
|  0|    1|
+---+-----+

 

Issue Links

• links to: GitHub Pull Request #43630