[SPARK-39405] NumPy Input Support in PySpark

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 3.4.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Shepherd: hyukjin.kwon
• Created: 07/Jun/22 18:22
• Updated: 11/Nov/22 20:04
• Resolved: 20/Sep/22 01:03

Description

NumPy is the fundamental package for scientific computing with Python. It is very commonly used, especially in the data science world. For example, Pandas is backed by NumPy, and Tensors also supports interchangeable conversion from/to NumPy arrays.

However, PySpark only supports Python native types with the exception of pandas data in “SparkSession.createDataFrame(pandas.DataFrame)”. “DataFrame.toPandas()”. NumPy input support has been requested multiple times, see SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857.

The ticket proposes to support NumPy input in PySpark.
 
See more design design.

Issue Links

• depends upon: SPARK-40084 Upgrade Py4J from 0.10.9.5 to 0.10.9.7
• depends upon: SPARK-40271 Support list type for pyspark.sql.functions.lit

Sub-Tasks

• Accept NumPy array in createDataFrame: Resolved
• Construct the schema from `np.dtype` when `createDataFrame` from a NumPy array: Resolved
• Accept a list that contains NumPy scalars in `createDataFrame`: Resolved
• Support NumPy scalars in built-in functions: Resolved
• Support NumPy ndarray in built-in functions: Resolved
• Consolidate `lit` function with NumPy scalar in sql and pandas module: Resolved
• DataFrame creation from 2d NumPy array with arbitrary columns: Resolved