[HIVE-28410] Partition-Aware Optimization for Iceberg or OTF

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Iceberg integration, StorageHandler
• Assignee: Shohei Okumiya
• Reporter: Shohei Okumiya
• Created: 28/Jul/24 06:23
• Updated: 01/Nov/24 10:41

Description

This project aims to allow users using Apache Iceberg or other non-native table formats to take advantage of Hive's advanced optimizations.

 

Apache Hive provides some optimizations depending on the storage layout of Hive native tables. Bucket Map Join, Sort Merge Bucket Join, or GroupByOptimizer are some of them. Those optimizations are not open to non-native tables because they rely on a piece of hardcoded logic. For example, hashing algorithms are implemented outside StorageHandlers, so enabling Bucket Map Join on Iceberg tables with Bucket Transforms is unrealistic.

 

We have some discussions in HIVE-27734; this is the first design doc.

Issue Links

• is duplicated by: HIVE-27734 Add Iceberg's storage-partitioned join capabilities to Hive's [sorted-]bucket-map-join

Sub-Tasks

• Bucket Map Join on Iceberg tables: Resolved
• Sort Merge Bucket Join for Iceberg tables: Open
• Group By Optimization for Iceberg tables: Open
• Flexible join optimization based on partition transform specs: Open