[SPARK-44076] SPIP: Python Data Source API

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Allison Wang
• Shepherd: Hyukjin Kwon
• Created: 16/Jun/23 04:05
• Updated: 10/Oct/23 18:20

Description

This proposal aims to introduce a simple API in Python for Data Sources. The idea is to enable Python developers to create data sources without having to learn Scala or deal with the complexities of the current data source APIs. The goal is to make a Python-based API that is simple and easy to use, thus making Spark more accessible to the wider Python developer community. This proposed approach is based on the recently introduced Python user-defined table functions (SPARK-43797) with extensions to support data sources.

SPIP: https://docs.google.com/document/d/1oYrCKEKHzznljYfJO4kx5K_Npcgt1Slyfph3NEk7JRU/edit?usp=sharing

Sub-Tasks

• Initial support for Python data source read API: Resolved
• Support registering Python data sources: Resolved
• Support loading Python data sources in DataFrameReader: Resolved
• Add InputPartition to DataSourceReader interface: Resolved
• Add Python data source write API: Resolved
• Make Python data source registration session level: Resolved
• Plan Python data source read using mapInArrow: Resolved
• Change saveMode to overwrite for DataSourceWriter constructor: Resolved
• Support spark.read.schema(...) for Python data source API: Resolved
• Respect column names when Python data source read function outputs named Row objects: Resolved
• Initial support for Python data source write API: Resolved
• Support spark.read.load() with non-empty path for Python data source API: Open
• Support creating table using a Python data source in SQL: Resolved
• Support `commit` and `abort` API for Python data source write: Resolved
• Support overwrite mode for Python data source write: Resolved
• Investigate runtime registration and feasibility of overwriting the datasource: Resolved
• Statically register Python Data Source: Resolved
• Update `path` handling in Python data source: Resolved
• Allow non-deterministic Python UDFs in MapInPandas/MapInArrow: Resolved
• Support create table using DSv2 sources: Resolved
• Support CTAS using DSv2 sources: Resolved
• Support INSERT INTO/OVERWRITE using DSv2 sources: Resolved
• Add documentation for Python data source API: Resolved
• Refactor Python Data Source instance loading: Resolved
• Support PythonSQLMetrics.pythonMetrics: Resolved
• Add a new API in DSv2 DataWriter to write an iterator of records: Resolved
• Block Python data source registration with name conflicts: Resolved
• Improve error messages for invalid save mode: Resolved
• Check Python executable when looking up available Data Sources: Resolved
• Improve Python data source error classes and messages: Resolved
• Python data source options should be a case insensitive dictionary: Resolved
• Improve error messages for unsupported data source save mode: Resolved
• Log full exception when failed to lookup Python Data Sources: Resolved
• Disallow re-registration of statically registered data sources: Open
• Improve error messages for DATA_SOURCE_NOT_FOUND error: Resolved
• Make DataSourceManager isolated and self clone-able: Resolved
• Refactor Python Data Source to align with other built-in Data Sources: Resolved
• Skip test_datasource if PyArrow is not installed: Resolved
• Skip V2 table lookup when a table is in V1 table cache: Resolved
• Make daemon mode configurable when creating Python workers: Resolved
• Support Python data source API with Spark Connect: Resolved
• Fix docstring links and type hints in Python Data Source: Resolved
• Document Python Data Source API in API reference page: Resolved
• Remove the private[sql] modifier for Python data sources: Resolved