[HIVE-27409] Iceberg: table with EXTERNAL type can not use statistics to optimize the query

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: Iceberg integration
• Labels: pull-request-available
• Assignee: Butao Zhang
• Reporter: Butao Zhang
• Created: 04/Jun/23 15:07
• Updated: 15/Aug/23 06:33
• Resolved: 07/Jun/23 09:27

Description

We have supported iceberg statistics recently. e.g. HIVE-24928  and  HIVE-27158. And we can use iceberg stats to optimize some queries like HIVE-27347.

However, in current hive codebase, we prohibit using EXTERNAL table stats and this change was introduced by HIVE-11266. And HIVE-19329 also disabled some optimizations for EXTERNAL table whether it is iceberg or not.  Therefore, The EXTERNAL type iceberg table can not use stats to optimize query.

 

In HIVE-24928,  we have added method HiveStorageHandler::canProvideBasicStatistics()  to indicate iceberg can have the ability to provide stats. That is to say, Although Iceberg table is regard as EXTERNAL table in Hive, it can provide details statistics.

 

Therefore, here i suggest we should check both table type and boolean result of HiveStorageHandler::canProvideBasicStatistics() to determine if the table can use stats.

Issue Links

• relates to: HIVE-19329 Disallow some optimizations/behaviors for external tables
• relates to: HIVE-11266 count(*) wrong result based on table statistics for external tables
• relates to: HIVE-24928 In case of non-native tables use basic statistics from HiveStorageHandler
• links to: GitHub Pull Request #4388
• links to: GitHub Pull Request #4395