[HIVE-19329] Disallow some optimizations/behaviors for external tables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 26/Apr/18 21:03
• Updated: 05/Jun/23 02:46

Description

External tables in Hive are often used in situations where the data is being created and managed by other applications outside of Hive. There are several issues that can occur when data being written to table directories by external apps:

• If an application is writing files to a table/partition at the same time that Hive tries to merge files for the same table/partition (ALTER TABLE CONCATENATE, or hive.merge.tezfiles during insert) data can be lost.
• When new data has been added to the table by external applications, the Hive table statistics are often way out of date with the current state of the data. This can result in wrong results in the case of answering queries using stats, or bad query plans being generated.

Some of these operations should be blocked in Hive. It looks like some already have been (HIVE-17403).

Issue Links

• is related to: HIVE-27409 Iceberg: table with EXTERNAL type can not use statistics to optimize the query

Sub-Tasks

• Disable compute.query.using.stats for external table: Closed
• Disable condition tree branch removal using stats for external tables: Open
• Use actual file size rather than stats for fetch task optimization with external tables: Closed
• Disable runtime filtering (semijoin reduction opt with bloomfilter) for external tables: Closed
• Disable SMB/Bucketmap join for external tables: Closed
• Disable stats autogather for external tables: Open