[HIVE-23553] Upgrade ORC version to 1.6.7

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 27/May/20 09:19
• Updated: 17/Apr/24 12:13
• Resolved: 04/Feb/21 00:04

Description

 Apache Hive is currently on 1.5.X version and in order to take advantage of the latest ORC improvements such as column encryption we have to bump to 1.6.X.

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288&styleName=&projectId=12318320&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin

Even though ORC reader could work out of the box, HIVE LLAP is heavily depending on internal ORC APIs e.g., to retrieve and store File Footers, Tails, streams – un/compress RG data etc. As there ware many internal changes from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the upgrade is not straightforward.

This Umbrella Jira tracks this upgrade effort.

Issue Links

• blocks: HIVE-22731 MJ probe decode with row-level filtering
• causes: HIVE-24721 Align CacheWriter bufferSize with Llap max alloc
• is duplicated by: HIVE-24536 Upgrade ORC to 1.6.6
• is fixed by: HIVE-28202 Incorrect projected column size after ORC upgrade to v1.6.7
• relates to: HIVE-24735 Implement TIMESTAMP WITH LOCAL TIME ZONE integration with ORC
• relates to: HIVE-24631 Move to ChronoLocalDate and day of epoch instead of java's sql Date
• relates to: HIVE-24703 LLAP move RowGroup reading to StrippePlanner
• links to: GitHub Pull Request #1823

Sub-Tasks

• DiskRangeInfo should make use of DiskRangeList: Resolved
• ORC PPD eval with sub-millisecond timestamps: Closed