[HIVE-19027] Make materializations invalidation cache work with multiple active remote metastores

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Materialized views
• Target Version/s: 3.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 22/Mar/18 21:03
• Updated: 27/Feb/24 22:23
• Resolved: 27/Jun/18 07:34

Description

The main points:

• Only MVs that use transactional tables can have a time window value of 0. Those are the only MVs that can be guaranteed to not be outdated when a query is executed.
• For MVs that cannot be outdated, comparison is based on valid write id lists.
• For MVs that can be outdated:
  • The window for valid outdated MVs can be specified in intervals of 1 minute.
  • A materialized view is outdated if it was built before that time window and any source table has been modified since.

A time window of -1 means to always use the materialized view for rewriting without any checks concerning its validity. If a materialized view uses an external table, the only way to trigger the rewriting would be to set the property to -1, since currently we do not capture for validation purposes whether the external source tables have been modified since the MV was created or not.

Issue Links

• incorporates: HIVE-19409 Disable incremental rewriting with outdated materialized views
• is blocked by: HIVE-19017 Add util function to determine if 2 ValidWriteIdLists are at the same committed ID
• is blocked by: HIVE-18825 Define ValidTxnList before starting query optimization
• is cloned by: HIVE-20006 Make materializations invalidation cache work with multiple active remote metastores
• relates to: HIVE-18667 Materialized views: rewrites should be triggered without checks if the time.window=-1
• links to: RB link