[IMPALA-2716] Hive/Impala incompatibility for timestamp data in Parquet

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: incompatibility parquet
• Target Version: Impala 2.9.0
• Assignee: Attila Jeges
• Reporter: Alexander Behm
• Created: 26/Nov/15 01:26
• Updated: 04/May/17 08:17
• Resolved: 04/May/17 08:17

Description

Problem
Hive adjusts timestamps by subtracting the local time zone’s offset from all values when writing data to Parquet files. Hive is internally inconsistent because it behaves differently for other file formats. As a result of this adjustment, Impala may read "incorrect" timestamp values from Parquet files written by Hive, and vice versa.

Workaround
Enable the following compatibility flag in Impala which is false by default.

--convert_legacy_hive_parquet_utc_timestamps
When true, TIMESTAMPs read from files written by Parquet-MR (used by Hive) will be converted from UTC to local time. Writes are unaffected.

For more details, please see IMPALA-1658