[HIVE-21029] External table replication for existing deployments running incremental replication.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.0.0, 3.1.1
• Fix Version/s: 4.0.0-alpha-1
• Component/s: repl
• Labels: pull-request-available
• Assignee: Sankar Hariappan
• Reporter: Anishek Agarwal
• Created: 11/Dec/18 06:05
• Updated: 17/Nov/22 08:55
• Resolved: 31/Jan/19 17:11

Description

Existing deployments using hive replication do not get external tables replicated. For such deployments to enable external table replication they will have to provide a specific switch to first bootstrap external tables as part of hive incremental replication, following which the incremental replication will take care of further changes in external tables.

The switch will be provided by an additional hive configuration (for ex: hive.repl.bootstrap.external.tables) and is to be used in

 WITH 

clause of

 REPL DUMP 

command.

Additionally the existing hive config hive.repl.include.external.tables will always have to be set to "true" in the above clause.

Proposed usage for enabling external tables replication on existing replication policy.
1. Consider an ongoing repl policy <db1> in incremental phase.
Enable hive.repl.include.external.tables=true and hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP.

• Dumps all events but skips events related to external tables.
• Instead, combine bootstrap dump for all external tables under “_bootstrap” directory.
• Also, includes the data locations file "_external_tables_info”.
• LIMIT or TO clause shouldn’t be there to ensure the latest events are dumped before bootstrap dumping external tables.

2. REPL LOAD on this dump applies all the events first, copies external tables data and then bootstrap external tables (metadata).

• It is possible that the external tables (metadata) are not point-in time consistent with rest of the tables.
• But, it would be eventually consistent when the next incremental load is applied.
• This REPL LOAD is fault tolerant and can be retried if failed.

3. All future REPL DUMPs on this repl policy should set hive.repl.bootstrap.external.tables=false.

• If not set to false, then target might end up having inconsistent set of external tables as bootstrap wouldn’t clean-up any dropped external tables.

Issue Links

• relates to: HIVE-21286 Hive should support clean-up of previously bootstrapped tables when retry from different dump.
• links to: GitHub Pull Request #523