[HIVE-16865] Handle replication bootstrap of large databases

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: HiveServer2
• Assignee: Anishek Agarwal Assign to me
• Reporter: Anishek Agarwal
• Created: 09/Jun/17 05:51
• Updated: 21/Oct/22 07:20

Description

for larger databases make sure that we can handle replication bootstrap.

• Assuming large database can have close to million tables or a few tables with few hundred thousand partitions.

• for function replication if a primary warehouse has large number of custom functions defined such that the same binary file in corporates most of these functions then on the replica warehouse there might be a problem in loading all these functions as we will have the same jar on primary copied over for each function such that each function will have a local copy of the jar, loading all these jars might lead to excessive memory usage.

Sub-Tasks

• Move creation of _files from ReplCopyTask to analysis phase for boostrap replication: Closed
• Multi-threaded execution of bootstrap dump of tables / functions: Open
• move replication dump related work in semantic analysis phase to execution phase using a task: Closed
• Multi-threaded execution of bootstrap dump of partitions: Closed
• move replication load related work in semantic analysis phase to execution phase using a task: Closed
• repl load does not lead to excessive memory consumption for multiple functions from same binary jar: Open
• during repl load for large number of partitions the metadata file can be huge and can lead to out of memory: Open
• ability to trace the task tree created during repl load from logs: Open
• Incremental replication load should create tasks in execution phase rather than semantic phase: Closed