[HIVE-20006] Make materializations invalidation cache work with multiple active remote metastores

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Materialized views
• Target Version/s: 4.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 27/Jun/18 06:31
• Updated: 27/Feb/24 22:23
• Resolved: 14/Jul/18 06:08

Description

The main points:

• Only MVs that use transactional tables can have a time window value of 0. Those are the only MVs that can be guaranteed to not be outdated when a query is executed.
• For MVs that cannot be outdated, comparison is based on valid write id lists.
• For MVs that can be outdated:
  • The window for valid outdated MVs can be specified in intervals of 1 minute.
  • A materialized view is outdated if it was built before that time window and any source table has been modified since.

A time window of -1 means to always use the materialized view for rewriting without any checks concerning its validity. If a materialized view uses an external table, the only way to trigger the rewriting would be to set the property to -1, since currently we do not capture for validation purposes whether the external source tables have been modified since the MV was created or not.

Issue Links

• incorporates: HIVE-19409 Disable incremental rewriting with outdated materialized views
• is a clone of: HIVE-19027 Make materializations invalidation cache work with multiple active remote metastores
• is blocked by: HIVE-19017 Add util function to determine if 2 ValidWriteIdLists are at the same committed ID
• is blocked by: HIVE-18825 Define ValidTxnList before starting query optimization
• relates to: HIVE-18667 Materialized views: rewrites should be triggered without checks if the time.window=-1
• links to: RB link