[HIVE-21763] Incremental replication to allow changing include/exclude tables list in replication policy.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: repl
• Labels: DR Replication pull-request-available
• Target Version/s: 4.0.0
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 21/May/19 07:36
• Updated: 17/Nov/22 08:50
• Resolved: 19/Jun/19 03:18

Description

• REPL DUMP takes 2 inputs along with existing FROM and WITH clause.

  - REPL DUMP <current_repl_policy> [REPLACE <previous_repl_policy> FROM <last_repl_id> WITH <key_values_list>;
  - current_repl_policy and previous_repl_policy can be any format mentioned in Point-4.
  - REPLACE clause to be supported to take previous repl policy as input. If REPLACE clause is not there, then the policy remains unchanged.
  - Rest of the format remains same.
  

• Now, REPL DUMP on this DB will replicate the tables based on current_repl_policy.
• Single table replication of format <db_name>.t1 doesn’t allow changing the policy dynamically. So REPLACE clause is not allowed if previous_repl_policy of this format.
• If any table is added dynamically either due to change in regular expression or added to include list should be bootstrapped using independant table level replication policy.

  - Hive will automatically figure out the list of tables newly included in the list by comparing the current_repl_policy & previous_repl_policy inputs and combine bootstrap dump for added tables as part of incremental dump. "_bootstrap" directory can be created in dump dir to accommodate all tables to be bootstrapped.
  - If any table is renamed, then it may gets dynamically added/removed for replication based on defined replication policy + include/exclude list. So, Hive will perform bootstrap for the table which is just included after rename.
  

• REPL LOAD should check for changes in repl policy and drop the tables/views excluded in the new policy compared to previous policy. It should be done before performing incremental and bootstrap load from the current dump.
• REPL LOAD on incremental dump should load events directories first and then check for "_bootstrap" directory and perform bootstrap load on them.

Rename table is not in scope of this jira.

Issue Links

• relates to: HIVE-21893 Handle concurrent write + drop when ACID tables are getting bootstrapped.
• links to: GitHub Pull Request #673