[SPARK-35999] Make from_csv/to_csv to handle day-time intervals properly

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.3.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 02/Jul/21 19:42
• Updated: 06/Jul/21 09:39
• Resolved: 06/Jul/21 09:39

Description

from_csv throws exception if day-time interval types are given.

spark-sql> select from_csv("interval '1 2:3:4' day to second", "a interval day to second");
21/07/03 04:39:13 ERROR SparkSQLDriver: Failed in [select from_csv("interval '1 2:3:4' day to second", "a interval day to second")]
java.lang.Exception: Unsupported type: interval day to second
 at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedTypeError(QueryExecutionErrors.scala:775)
 at org.apache.spark.sql.catalyst.csv.UnivocityParser.makeConverter(UnivocityParser.scala:224)
 at org.apache.spark.sql.catalyst.csv.UnivocityParser.$anonfun$valueConverters$1(UnivocityParser.scala:134)
 

Also, to_csv doesn't handle day-time interval types properly though any exception is thrown.
The result of to_csv for day-time interval types is not ANSI interval compliant form.

spark-sql> select to_csv(named_struct("a", interval '1 2:3:4' day to second));
93784000000

The result above should be INTERVAL '1 02:03:04' DAY TO SECOND.

Issue Links

• links to: [Github] Pull Request #33226 (sarutak)
• links to: [Github] Pull Request #33226 (sarutak)