[HIVE-24074] Incorrect handling of timestamp in Parquet/Avro when written in certain time zones in versions before Hive 3.x

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Avro, Parquet
• Labels: compatibility pull-request-available timestamp
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 26/Aug/20 04:21
• Updated: 27/Feb/24 22:24
• Resolved: 28/Aug/20 15:18

Description

The timezone conversion for Parquet and Avro uses new java.time.* classes, which can lead to incorrect values returned for certain dates in certain timezones if timestamp was computed and converted based on java.sql.* classes. For instance, the offset used for Singapore timezone in 1900-01-01T00:00:00.000 is UTC+8, while the correct offset for that date should be UTC+6:55:25. Some additional information can be found here: https://stackoverflow.com/a/52152315

Issue Links

• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release
• is depended upon by: HIVE-26233 Problems reading back PARQUET timestamps above 10000 years
• is related to: HIVE-25104 Backward incompatible timestamp serialization in Parquet for certain timezones
• is related to: HIVE-26270 Wrong timestamps when reading Hive 3.1.x Parquet files with vectorized reader
• is related to: HIVE-26650 Run qtests with different default encoding
• links to: GitHub Pull Request #1392