Parquet files written in Hive 3.1.x onwards with timezone set to US/Pacific.
Parquet files read with Hive 4.0.0-apha-1 onwards.
results are correct.
some timestamps are shifted.
The problem is the same reported under
HIVE-24074. The data were written using the new Date/Time APIs (java.time) in version Hive 3.1.3 and here they were read using the old APIs (java.sql).
The difference with
HIVE-24074 is that here the problem appears only for vectorized execution while the non-vectorized reader is working fine so there is some inconsistency in the behavior of vectorized and non vectorized readers.
Non-vectorized reader works fine cause it derives automatically that it should use the new JDK APIs to read back the timestamp value. This is possible in this case cause there are metadata information in the file (i.e., the presence of writer.time.zone) from where it can infer that the timestamps were written using the new Date/Time APIs.
The inconsistent behavior between vectorized and non-vectorized reader is a regression caused by
HIVE-25104. This JIRA is an attempt to re-align the behavior between vectorized and non-vectorized readers.
Note that if the file metadata are empty both vectorized and non-vectorized reader cannot determine which APIs to use for the conversion and in this case it is necessary the user to set the
hive.parquet.timestamp.legacy.conversion.enabled explicitly to get back the correct results.