[SPARK-39494] Support `createDataFrame` from a list of scalars when schema is not provided

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Xinrong Meng
• Created: 16/Jun/22 21:46
• Updated: 20/Sep/22 16:49

Description

Currently, DataFrame creation from a list of native Python scalars is unsupported in PySpark, for example,

>>> spark.createDataFrame([1, 2]).collect()
Traceback (most recent call last):
...
TypeError: Can not infer schema for type: <class 'int'>

However, Spark DataFrame Scala API supports that:

scala> Seq(1, 2).toDF().collect()
res6: Array[org.apache.spark.sql.Row] = Array([1], [2])

To maintain API consistency, we propose to support DataFrame creation from a list of scalars. 

See more [here](https://docs.google.com/document/d/1Rd20PVbVxNrLfOmDtetVRxkgJQhgAAtJp6XAAZfGQgc/edit?usp=sharing).

Issue Links

• links to: [Github] Pull Request #36893 (xinrong-databricks)
• links to: [Github] Pull Request #36893 (xinrong-databricks)