[SPARK-16216] CSV data source does not write date and timestamp correctly

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Labels: releasenotes
• Target Version/s: 2.0.1, 2.1.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 26/Jun/16 05:54
• Updated: 12/Dec/22 18:11
• Resolved: 25/Aug/16 04:19

Description

Currently, CSV data source write DateType and TimestampType as below:

+----------------+
|            date|
+----------------+
|1440637200000000|
|1414459800000000|
|1454040000000000|
+----------------+

It would be nicer if it write dates and timestamps as a formatted string just like JSON data sources.

Also, CSV data source currently supports dateFormat option to read dates and timestamps in a custom format. It might be better if this option can be applied in writing as well.

Issue Links

• duplicates: SPARK-16597 DataFrame DateType is written as an int(Days since epoch) by csv writer
• is duplicated by: SPARK-17066 dateFormat should be used when writing dataframes as csv files
• is duplicated by: SPARK-17923 dateFormat unexpected kwarg to df.write.csv
• links to: [Github] Pull Request #13912 (HyukjinKwon)
• links to: [Github] Pull Request #14279 (HyukjinKwon)
• links to: [Github] Pull Request #14799 (HyukjinKwon)
• links to: [Github] Pull Request #14829 (HyukjinKwon)
• links to: [Github] Pull Request #14840 (HyukjinKwon)