[HIVE-25656] Get materialized view state based on number of affected rows of transactions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Materialized views, Thrift API, Transactions
• Labels: pull-request-available
• Assignee: Krisztian Kasa
• Reporter: Krisztian Kasa
• Created: 28/Oct/21 08:24
• Updated: 05/Apr/24 11:58
• Resolved: 23/Nov/21 18:44

Description

To enable the faster incremental rebuild of materialized views presence of update/delete operations on the source tables of the view since the last rebuild must be checked. Based on the outcome different plan is generated for scenarios in presence of update/delete and insert only operations.

Currently this is done by querying the COMPLETED_TXN_COMPONENTS table however the records from this table is cleaned when MV source tables are compacted. This reduces the chances of incremental MV rebuild.

The goal of this patch is to find an alternative way to store and retrieve this information.

Issue Links

• is related to: HIVE-25878 Unable to compile cpp metastore thrift client
• is related to: HIVE-26375 Invalid materialized view after rebuild if source table was compacted
• is related to: HIVE-25745 Print transactional stats of materialized view source tables
• is related to: HIVE-25747 Make a cost base decision when rebuilding materialized views
• is related to: HIVE-25744 Support backward compatibility of thrift struct CreationMetadata
• is required by: IMPALA-11390 Describe formatted statement on materialized view should show the view definition
• links to: GitHub Pull Request #2756