Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16216

CSV data source does not write date and timestamp correctly

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.1, 2.1.0
    • Component/s: SQL
    • Labels:

      Description

      Currently, CSV data source write DateType and TimestampType as below:

      +----------------+
      |            date|
      +----------------+
      |1440637200000000|
      |1414459800000000|
      |1454040000000000|
      +----------------+
      

      It would be nicer if it write dates and timestamps as a formatted string just like JSON data sources.

      Also, CSV data source currently supports dateFormat option to read dates and timestamps in a custom format. It might be better if this option can be applied in writing as well.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: