[SPARK-33055] Add support for CalendarIntervalType in PySpark schema

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: PySpark, SQL
• Assignee: Unassigned
• Reporter: Maciej Szymkiewicz
• Created: 02/Oct/20 15:13
• Updated: 02/Oct/20 15:29

Description

Queries resulting in interval types shouldn't fail in PySpark, i.e. the following

from pyspark.sql.functions import current_timestamp                                                                                                                                  

spark.range(1).select(current_timestamp() - current_timestamp())    

should result in

DataFrame[(current_timestamp() - current_timestamp()): interval]

instead of ValueError.

Issue Links

• links to: [Github] Pull Request #29935 (zero323)