[SPARK-37697] Make it easier to convert numpy arrays to Spark Dataframes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Douglas Moore
• Created: 20/Dec/21 16:50
• Updated: 02/Nov/22 21:06

Description

Make it easier to convert numpy arrays to dataframes.

Often we receive errors:

 

df = spark.createDataFrame(numpy.arange(10))
Can not infer schema for type: <class 'numpy.int64'>

 

OR

df = spark.createDataFrame(numpy.arange(10.))
Can not infer schema for type: <class 'numpy.float64'>

 

Today (Spark 3.x) we have to:

spark.createDataFrame(pd.DataFrame(numpy.arange(10.))) 

Make this easier with a direct conversion from Numpy arrays to Spark Dataframes.