[HIVE-18839] Implement incremental rebuild for materialized views (only insert operations in source tables)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Materialized views
• Labels: TODOC3.0
• Target Version/s: 3.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 01/Mar/18 23:40
• Updated: 27/Feb/24 22:24
• Resolved: 10/Apr/18 09:33

Description

Implementation will follow current code path for full rebuild.

When the MV query plan is retrieved, if the MV contents are outdated because there were insert operations in the source tables, we will introduce a filter with a condition based on stored value of ValidWriteIdLists. For instance, WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...). Then the rewriting will do the rest of the work by creating a partial rewriting, where the contents of the MV are read as well as the new contents from the source tables.

This mechanism will not work only for ALTER MV... REBUILD, but also for user queries which will be able to benefit from using outdated MVs to compute part of the needed results.

Issue Links

• causes: HIVE-19114 MV rewriting not being triggered for last query in materialized_view_rewrite_4.q
• contains: HIVE-18667 Materialized views: rewrites should be triggered without checks if the time.window=-1
• is blocked by: HIVE-18957 Upgrade Calcite version to 1.16.0
• is depended upon by: HIVE-19111 ACID: PPD & Split-pruning for txn_id filters
• relates to: HIVE-24854 Incremental Materialized view refresh in presence of update/delete operations
• links to: RB link