[HIVE-28202] Incorrect projected column size after ORC upgrade to v1.6.7

• Type: Bug
• Status: Patch Available
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.0.0, 4.0.0-beta-1
• Fix Version/s: 4.1.0
• Component/s: Hive
• Labels: hive-4.0.1-must performance pull-request-available
• Assignee: Denys Kuzmenko
• Reporter: Denys Kuzmenko
• Created: 17/Apr/24 11:42
• Updated: 20/Apr/24 20:07

Description

`ReaderImpl.getRawDataSizeFromColIndices` changed behavior for handling struct type and now includes their subtypes. That caused an issue in Hive as the root struct index is always "included", causing size estimation for the complete schema, not just selected columns leading to incorrect split estimations.

Issue Links

• fixes: HIVE-23553 Upgrade ORC version to 1.6.7
• links to: GitHub Pull Request #5195