[HIVE-20332] Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Materialized views
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 07/Aug/18 18:03
• Updated: 27/Feb/24 22:24
• Resolved: 14/Aug/18 05:54

Description

Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this should be fixed by HIVE-20313). Even if we did, we always assume uniform distribution of the column values, which can easily lead to overestimations on the number of rows read when we filter on ROW__ID.writeId for materialized views (think about a large transaction for MV creation and then small ones for incremental maintenance). This overestimation can lead to incremental view maintenance not being triggered as cost of the incremental plan is overestimated (we think we will read more rows than we actually do). This could be fixed by introducing histograms that reflect better the column values distribution.

Till both fixes are implemented, we will use a config variable that will set the selectivity for filter condition on ROW__ID during the cost calculation. Setting that variable to a low value will favour incremental rebuild over full rebuild.

Issue Links

• is superceded by: HIVE-20382 Materialized views: Introduce heuristic to favour incremental rebuild
• relates to: HIVE-20313 consider making ROW__ID a 1st class object
• links to: RB link