[HIVE-19499] Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0, 4.0.0-alpha-1
• Component/s: HiveServer2, repl
• Labels: DR pull-request-available replication
• Target Version/s: 3.1.0, 4.0.0
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 11/May/18 10:30
• Updated: 21/Oct/22 06:59
• Resolved: 31/May/18 04:32

Description

Currently. bootstrap REPL LOAD expect the target database to be empty or not exist to start bootstrap load.

But, this adds overhead when there is a failure in between bootstrap load and there is no way to resume it from where it fails. So, it is needed to create checkpoints in table/partitions to skip the completely loaded objects.

Use the fully qualified path of the dump directory as a checkpoint identifier. This should be added to the table / partition properties in hive via a task, as the last task in the DAG for table / partition creation.

Issue Links

• relates to: HIVE-19739 Bootstrap REPL LOAD to use checkpoints to validate and skip the loaded data/metadata.
• links to: GitHub Pull Request #352