[HIVE-21886] REPL - With table list - Handle rename events during replace policy

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: repl
• Labels: DR Replication pull-request-available
• Target Version/s: 4.0.0
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Created: 18/Jun/19 08:23
• Updated: 16/Jun/20 00:24
• Resolved: 30/Jun/19 11:00

Description

If some rename events are found to be dumped and replayed while replace policy is getting executed, it needs to take care of the policy inclusion in both the policy for each table name.

 1. Create a list of tables to be bootstrapped. 

  2. During handling of alter table, if the alter type is rename 

      1. If the old table name is present in the list of table to be bootstrapped, remove it.

       2. If the new table name, matches the new policy, add it to the list of tables to be bootstrapped.

       3. If the old table does not match the old policy drop it, even if the table is not present at target.

  3. During handling of drop table

       1. if the table is in the list of tables to be bootstrapped, then remove it and ignore the event.

  4. During other event handling 

       1. if the table is there in the list of tables to be bootstrapped, then ignore the event.

       2. If the new policy does not match the table name, then ignore the event.

 

Rename handling during replace policy

1. Old name not matching old policy – The old table will not be there at the target cluster. The table will not be returned by get-all-table.
   1. Old name is not matching new policy
      1. New name not matching old policy
         1. New name not matching new policy
            • Ignore the event, no need to do anything.
         2. New name matching new policy
            • The table will be returned by get-all-table. Replace policy handler will bootstrap this table as its matching new policy and not matching old policy.
            • All the future events will be ignored as part of check added by replace policy handling.
            • All the event with old table name will anyways be ignored as the old name is not matching the new policy.
      2. New name matching old policy
         1. New name not matching new policy
            • As the new name is not matching the new policy, the table need not be replicated.
            • As the old name is not matching the new policy, the rename events will be ignored.
            • So nothing to be done for this scenario.
         2. New name matching new policy
            • As the new name is matching both old and new policy, replace handler will not bootstrap the table.
            • Add the table to the list of tables to be bootstrapped.
            • Ignore all the events with new name.
            • If there is a drop event for the table (with new name), then remove the table from the the list of table to be bootstrapped.
            • In case of rename event (double rename)
              • If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
              • If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped.
   2. Old name is matching new policy – As per replace policy handler, which checks based on old table, the table should be bootstrapped and event should be ignored. But rename handler should decide based on new name.The old table name will not be returned by get-all-table, so replace handler will not d anything for the old table.
      1. New name not matching old policy
         1. New name not matching new policy
            • As the old table is not there at target and new name is not matching new policy. Ignore the event.
            • No need to add the table to the list of tables to be bootstrapped.
            • All the subsequent events will be ignored as the new name is not matching the new policy.
         2. New name matching new policy
            • As the new name is not matching old policy but matching new policy, the table will be bootstrapped by replace policy handler. So rename event need not add this table to list of table to be bootstrapped.
            • All the future events will be ignored by replace policy handler.
            • For rename event (double rename)
              • If there is a rename, the table (with intermittent new name) will not be present and thus replace handler will not bootstrap the table.
              • So if the new name (the latest one) is matching the new policy, then add it to the list of table to be bootstrapped.
              • And If the new name (the latest one)  is not matching the new policy, then just ignore the event as the  intermittent new name would not have added to the list of table to be bootstrapped.
      2. New name matching old policy
         1. New name not matching new policy
            • Dump the event. The table will be dropped by repl load at the target.
         2. New name matching new policy
            • Replace handler will not bootstrap this table as the new name is matching both policies.
            • As old name is not matching the old policy, the table will not be there at target. The rename event should add the new name to the list of table to be bootstrapped.
            • Subsequent events with new table name should be ignored.
            • Drop events should not be ignored as if the table is present during bootstrapped, then its a new table and thus should be dropped.
            • In case of rename event (double rename)
              • If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
              • If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped.
2. Old name is matching old policy – The old table will be there at the target cluster. The table will not be returned by get-all-table. Repl load should delete the old table as it is not matching the new policy.
   1. Old name is not matching new policy
      1. New name not matching old policy
         1. New name not matching new policy
            • Nothing to be done. Ignore the event.
         2. New name matching new policy
            • As the new name is not matching old policy but matching new policy, the table will be bootstrapped by replace policy handler. So rename event need not add this table to list of table to be bootstrapped.
            • All the future events will be ignored by replace policy handler.
            • For rename event (double rename)
              • If there is a rename, the table (with intermittent new name) will not be present and thus replace handler will not bootstrap the table.
              • So if the new name (the latest one) is matching the new policy, then add it to the list of table to be bootstrapped.
              • And If the new name (the latest one)  is not matching the new policy, then just ignore the event as the  intermittent new name would not have added to the list of table to be bootstrapped.
      2. New name matching old policy
         1. New name not matching new policy
            • Table with new name will be dropped by repl load
            • Along with other event, ignore the rename event also.
         2. New name matching new policy
            • As the new name is matching both old and new policy, replace handler will not bootstrap the table.
            • Add the table to the list of tables to be bootstrapped.
            • Ignore all the events with new name.
            • If there is a drop event for the table (with new name), then remove the table from the the list of table to be bootstrapped.
            • In case of rename event (double rename)
              • If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
              • If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped.
   2. Old name is matching new policy
      1. New name not matching old policy
         1. New name not matching new policy
            • The old table needs to be dropped at target. Ignore this event, as the old table is not matching the new policy, it will be dropped by repl load.
         2. New name matching new policy
            • Allow the event to dump and replayed at target.
            • Allow further events to be handled as usual.
            • In case of rename event (double rename)
              • If the latest new name is matching the new policy, then keep it as is it. Let rename event replayed at target.
              • If the latest new name is not matching the new policy, then change the rename event to drop event.
      2. New name matching old policy
         1. New name not matching new policy
            1. Nothing to be done.
         2. New name matching new policy
            1. Add the table name to the list of tables to be bootstrapped.

Issue Links

• links to: GitHub Pull Request #688