[HIVE-22589] Add storage support for ProlepticCalendar in ORC, Parquet, and Avro

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Avro, ORC, Parquet
• Labels: compatibility datetime
• Assignee: Jesus Camacho Rodriguez
• Reporter: Jesus Camacho Rodriguez
• Created: 06/Dec/19 08:59
• Updated: 25/Jun/21 08:30
• Resolved: 19/Feb/20 17:36

Description

Hive recently moved its processing to the proleptic calendar, which has created some issues for users who have dates before 1580 AD.

HIVE-22405 extended the column vectors for times & dates to encode which calendar they are using.

This issue is to support proleptic calendar in ORC, Parquet, and Avro, when files are written/read by Hive. To preserve compatibility with other engines until they upgrade their readers, files will be written using hybrid calendar by default. Default behavior when files do not contain calendar information in their metadata is configurable.

Issue Links

• incorporates: HIVE-22667 ORC version upgrade including ORC-578
• is blocked by: ORC-578 IllegalArgumentException: Can't use LongColumnVector to read proleptic Gregorian dates.
• relates to: HIVE-22405 Add ColumnVector support for ProlepticCalendar