[SPARK-29904] Parse timestamps in microsecond precision by JSON/CSV datasources

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4
• Fix Version/s: 2.4.5
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 14/Nov/19 20:48
• Updated: 03/Feb/20 17:43
• Resolved: 15/Nov/19 23:42

Description

Currently, Spark can parse strings with timestamps from JSON/CSV in millisecond precision. Internally, timestamps have microsecond precision. The ticket aims to modify parsing logic in Spark 2.4 to support the microsecond precision. Porting of DateFormatter/TimestampFormatter from Spark 3.0-preview is risky, so, need to find another lighter solution.

Issue Links

• is related to: SPARK-27224 Spark to_json parses UTC timestamp incorrectly
• is related to: SPARK-6385 ISO 8601 timestamp parsing does not support arbitrary precision second fractions
• is related to: SPARK-10681 DateTimeUtils needs a method to parse string to SQL's timestamp value
• is required by: SPARK-29927 Parse timestamps in microsecond precision by `to_timestamp`, `to_unix_timestamp`, `unix_timestamp`
• links to: GitHub Pull Request #26507