[HIVE-25576] Configurable datetime formatter for unix_timestamp, from_unixtime

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0-beta-1
• Fix Version/s: 4.0.0
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Stamatis Zampetakis
• Reporter: Ashish Sharma
• Created: 29/Sep/21 12:20
• Updated: 29/Mar/24 17:49
• Resolved: 21/Aug/23 10:11

Description

HIVE-25403, HIVE-25458 switched the internal implementation of datetime formatter for unix_timestamp and from_unixtime from java.text.SimpleDateFormat to java.time.format.DateTimeFormatter in order fix some bugs and inconsistencies when the aforementioned functions are combined with other UDFs that have already migrated to use the new modern java.time package.

The two Java formatters present differences in their behavior leading to different query results. The supported patterns, between the two formatters, are also different something that makes existing queries crash at runtime (after upgrade). Adapting to the new behavior of DateTimeFormatter is a challenging and time-consuming task for end users especially due to the widespread use of the afforementioned unixtime functions.

Although DateTimeFormatter is a clear improvement over SimpleDateFormat some users still want to retain the old behavior for compatibility reasons thus introducing a property is necessary for facilitating migration.

The goal of this ticket is to introduce a new property namely hive.datetime.formatter to control the formatter used by unix_timestamp and from_unixtime. By default the new DateTimeFormatter is used while the use of SimpleDateFormat is discouraged. Eventually, SimpleDateFormat will cease to exist.

Issue Links

• is related to: HIVE-27673 Configurable datetime formatter for date_format
• relates to: HIVE-25403 Fix from_unixtime() to consider leap seconds
• relates to: HIVE-25306 Change Date/Timestamp parser from LENIENT to STRICT
• relates to: HIVE-25458 unix_timestamp() with string input give wrong result
• links to: GitHub Pull Request #2690
• links to: GitHub Pull Request #4615