[HIVE-14841] Replication - Phase 2

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Component/s: repl
• Assignee: Sushanth Sowmyan
• Reporter: Sushanth Sowmyan
• Created: 26/Sep/16 16:26
• Updated: 20/Dec/17 19:42

Description

Per email sent out to the dev list, the current implementation of replication in hive has certain drawbacks, for instance :

• Replication follows a rubberbanding pattern, wherein different tables/ptns can be in a different/mixed state on the destination, so that unless all events are caught up on, we do not have an equivalent warehouse. Thus, this only satisfies DR cases, not load balancing usecases, and the secondary warehouse is really only seen as a backup, rather than as a live warehouse that trails the primary.
• The base implementation is a naive implementation, and has several performance problems, including a large amount of duplication of data for subsequent events, as mentioned in HIVE-13348, having to copy out entire partitions/tables when just a delta of files might be sufficient/etc. Also, using EXPORT/IMPORT allows us a simple implementation, but at the cost of tons of temporary space, much of which is not actually applied at the destination.

Thus, to track this, we now create a new branch (repl2) and a uber-jira(this one) to track experimental development towards improvement of this situation.

Issue Links

• is a parent of: HIVE-16145 Replicate functions
• is a parent of: HIVE-18320 Support ACID Tables Replication
• relates to: HIVE-7973 Hive Replication Support
• relates to: HIVE-13836 DbNotifications giving an error = Invalid state. Transaction has already started
• relates to: HIVE-13966 DbNotificationListener: can loose DDL operation notifications
• relates to: HIVE-15232 Add notification events for functions and indexes
• links to: HiveReplicationv2Development

Sub-Tasks

• Improve the pathname returned by ReplicationSemanticAnalyzer.getNextDumpDir: Open
• Enhance REPL dump bootstrap to write out notifications that occurred while bootstrap was generating initial dump (implementing ReplicationSemanticAnalyzer.consolidateEvent): Open
• Move notification filtering to metastore server side: Open
• Repl rename support adds unnecessary duplication for non-rename alters: Open
• Optimize(reduce) the number of alter calls made to fix repl.last.id: Patch Available
• Bootstrap replication for constraint: Open
• Replicate views with proper query string when perform REPL LOAD on a renamed database.: Open
• Replicate materialized views creation metadata with correct database name: Open
• Support replication of Materialized views: Open