[SPARK-36490] Make from_csv/to_csv to handle timestamp_ntz type properly

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 12/Aug/21 07:25
• Updated: 13/Aug/21 09:09
• Resolved: 13/Aug/21 09:09

Description

In the current master, to_csv/from_csv can handle timestamp type like as follows.

SELECT to_csv(struct(TIMESTAMP"2021-11-23 11:22:33")); 
2021-11-23T11:22:33.000+09:00

SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP"); 
{"a":2021-11-23 11:22:33}

But they cannot handle timestamp_ntz type properly.

SELECT to_csv(struct(TIMESTAMP_NTZ"2021-11-23 11:22:33"));
-- 2021-11-23T11:22:33.000 is expected.
1637666553000000

SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP_NTZ");
21/08/12 16:12:49 ERROR SparkSQLDriver: Failed in [SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP_NTZ")]
java.lang.Exception: Unsupported type: timestamp_ntz
        at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedTypeError(QueryExecutionErrors.scala:777)
        at org.apache.spark.sql.catalyst.csv.UnivocityParser.makeConverter(UnivocityParser.scala:234)
        at org.apache.spark.sql.catalyst.csv.UnivocityParser.$anonfun$valueConverters$1(UnivocityParser.scala:134)

Issue Links

• links to: [Github] Pull Request #33719 (sarutak)