[SPARK-37275] Support ANSI intervals in PySpark

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: PySpark, SQL
• Labels: release-notes
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 11/Nov/21 03:24
• Updated: 12/Dec/22 18:10
• Resolved: 18/Nov/21 09:16

Description

This JIRA targets to implement ANSI interval types in PySpark:

• https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DayTimeIntervalType.scala
• https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/YearMonthIntervalType.scala

Issue Links

• contains: SPARK-37525 Timedelta support in pandas API on Spark

Sub-Tasks

• Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs: Resolved
• Support DayTimeIntervalType in Py4J: Resolved
• Support DayTimeIntervalType in Arrow: Resolved
• Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs: Resolved
• Support YearMonthIntervalType in Py4J: Resolved
• Support YearMonthIntervalType in Arrow: Resolved