[SPARK-30546] Make interval type more future-proof

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 17/Jan/20 08:12
• Updated: 02/Oct/20 15:42
• Resolved: 23/Jan/20 11:36

Description

We've decided to not follow the SQL standard to define the interval type in 3.0. We should try our best to hide intervals from data sources/external catalogs as much as possible, to not leak internals to external systems.

In Spark 2.4, intervals are exposed in the following places:
1. The `CalendarIntervalType` is public
2. `Colum.cast` accepts `CalendarIntervalType` and can cast string to interval.
3. `DataFrame.collect` can return `CalendarInterval` objects.
4. UDF can tale `CalendarInterval` as input.
5. data sources can return IntervalRow directly which may contain `CalendarInterval`.

In Spark 3.0, we don't want to break Spark 2.4 applications, but we should not expose intervals wider than 2.4. In general, we should avoid leaking intervals to DS v2 and catalog plugins. We should also revert some PostgresSQL specific interval features.

Issue Links

• relates to: SPARK-33054 Support interval type in PySpark

Sub-Tasks

• Add unstable annotation to the CalendarInterval class: Resolved
• Disable comparison for interval type: Resolved
• Invalidate interval type as a field table schema: Resolved
• Interval support for csv and json functions: Resolved
• Revert interval ISO/ANSI SQL Standard output since we decide not to follow ANSI and no round trip.: Resolved
• Expose the documentation of CalendarInternval in API documentation: Resolved