[SPARK-36078] Complete mappings between numpy literals and Spark data types

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Xinrong Meng
• Created: 09/Jul/21 18:23
• Updated: 09/Jul/21 18:24

Description

In https://github.com/apache/spark/pull/32955, the lit function defined in pyspark.pandas.spark.functions has been adjusted to support numpy literals input.

However, the mapping between numpy literals and Spark data types is not complete.

We ought to fill the gap.

Issue Links

• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark