[SPARK-31802] Format Java date-time types in Row.jsonValue directly

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 23/May/20 05:36
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/20 03:51

Description

Row.jsonValue has to convert incoming Java date/timestamps types to days/microseconds because DateFormatter/TimestampFormatter APIs didn't accept java.sql.Timestamp/java.util.Date and java.time.Instant/java.time.LocalDate before https://issues.apache.org/jira/browse/SPARK-31762. Internally, the formatters perform conversions to Java types again. This badly impacts on the performance. The ticket aims to add new APIs to DateFormatter and TimestampFormatter that should accept Java types.

Issue Links

• is a clone of: SPARK-31762 Fix perf regression of date/timestamp formatting in toHiveString
• links to: [Github] Pull Request #28620 (MaxGekk)
• links to: [Github] Pull Request #28620 (MaxGekk)