[IMPALA-8721] Wrong result when Impala reads a Hive written parquet TimeStamp column

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: Interoperability correctness hive impala parquet timestamp
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-9
• Assignee: Tim Armstrong
• Reporter: Abhishek Rawat
• Created: 27/Jun/19 18:53
• Updated: 13/Jul/23 15:21
• Resolved: 10/Feb/21 00:40

Description

 

Easy to repro on latest upstream:

hive> create table t1_hive(c1 timestamp) stored as parquet;
hive> insert into t1_hive values('2009-03-09 01:20:03.600000000');
hive> select * from t1_hive;
OK
2009-03-09 01:20:03.6
[localhost:21000] default> invalidate metadata t1_hive;
[localhost:21000] default> select * from t1_hive;
Query: select * from t1_hive
Query submitted at: 2019-06-24 09:55:36 (Coordinator: http://optimus-prime:25000)
Query progress can be monitored at: http://optimus-prime:25000/query_plan?query_id=b34f85cb5da29c26:d4dfcb2400000000
+-------------------------------+
| c1 |
+-------------------------------+
| 2009-03-09 09:20:03.600000000 | <<<<<UTC
+-------------------------------+
bin/start-impala-cluster.py --impalad_args='-convert_legacy_hive_parquet_utc_timestamps=true'
[localhost:21000] default> select * from t1_hive;
Query: select * from t1_hive
Query submitted at: 2019-06-24 10:00:22 (Coordinator: http://optimus-prime:25000)
Query progress can be monitored at: http://optimus-prime:25000/query_plan?query_id=d5428bb21fb259b9:7b10703400000000
+-------------------------------+
| c1 |
+-------------------------------+
| 2009-03-09 02:20:03.600000000 |. <<<<<<PST8PDT
+-------------------------------+
 

 

This issue is causing testcase test_hive_impala_interop to fail. Untill this issue is fixed, the testcase will be updated to not include a timestamp column. The test case should be updated to include a timestamp column once this issue is fixed.

 

Issue Links

• is related to: IMPALA-10491 Impala parquet scanner should use writer.time.zone when converting Hive timestamps
• relates to: HIVE-21290 Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
• relates to: IMPALA-1773 Implement TIMESTAMP WITH TIME ZONE data type