[SPARK-29949] JSON/CSV formats timestamps incorrectly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.4
• Fix Version/s: 2.4.5
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 18/Nov/19 17:49
• Updated: 19/Nov/19 09:11
• Resolved: 19/Nov/19 09:11

Description

For example:

scala> val t = java.sql.Timestamp.valueOf("2019-11-18 11:56:00.123456")
t: java.sql.Timestamp = 2019-11-18 11:56:00.123456
scala> Seq(t).toDF("t").select(to_json(struct($"t"), Map("timestampFormat" -> "yyyy-MM-dd HH:mm:ss.SSSSSS"))).show(false)
+-------------------------------------------------+
|structstojson(named_struct(NamePlaceholder(), t))|
+-------------------------------------------------+
|{"t":"2019-11-18 11:56:00.000123"}               |
+-------------------------------------------------+

Issue Links

• links to: GitHub Pull Request #26582