[SPARK-33054] Support interval type in PySpark

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: PySpark, SQL
• Assignee: Unassigned
• Reporter: Maciej Szymkiewicz
• Created: 02/Oct/20 15:10
• Updated: 02/Oct/20 15:42

Description

At the moment PySpark doesn't support interval types at all. For example calling the following

spark.sql("SELECT current_date() - current_date()")                                                                                                                                  

or

from pyspark.sql.functions import current_timestamp                                                                                                                                  
spark.range(1).select(current_timestamp() - current_timestamp())  

results in

Traceback (most recent call last):
...
ValueError: Could not parse datatype: interval

At minimum, we should support CalendarIntervalType in the schema, so queries using it don't fail on conversion.

Optionally we should have to provide conversions between internal and external types. That however, might be tricky, as CalendarInterval seems to have different semantics than datetime.timedetla.

Also see https://issues.apache.org/jira/browse/SPARK-21187?focusedCommentId=16474664&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16474664

Issue Links

• is related to: SPARK-30546 Make interval type more future-proof
• is related to: SPARK-28494 Expose CalendarIntervalType and CalendarInterval in Spark

Sub-Tasks

• Add support for CalendarIntervalType in PySpark schema: In Progress
• Add mapping between external Python types and CalendarIntervalType: Open