[HIVE-18320] Support ACID Tables Replication

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2, Metastore, repl, Transactions
• Labels: ACID DR replication
• Target Version/s: 4.0.0
• Assignee: Sankar Hariappan Assign to me
• Reporter: Sankar Hariappan
• Created: 20/Dec/17 19:35
• Updated: 17/Nov/22 08:54
• Resolved: 08/May/19 08:56

Description

Currently, Full ACID and MM (Micro-Managed) tables are not supported for replication.
Need to support it.

Issue Links

• is a child of: HIVE-14841 Replication - Phase 2

Sub-Tasks

• Introduce WriteID per table rather than using global transaction ID: Closed
• Migrate existing ACID tables to use write id per table rather than global transaction id: Closed
• create/replicate open transaction event: Closed
• Replicate Commit Txn operation (without writes): Closed
• Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.: Closed
• Rename table impacts the ACID behavior as table names are not updated in meta-tables.: Closed
• Need to replace transactionId with writeId in RecordIdentifier and other relevant contexts.: Closed
• Exchange partition should be disabled on ACID/Insert-only tables with per table write ID.: Closed
• ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.: Closed
• Correct methods and variables names which uses writeId instead of transactionId.: Closed
• ValidWriteIdList config should be defined on tables which has to collect stats after insert: Closed
• Create/Replicate Open, Commit (without writes) and Abort Txn events: Closed
• ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.: Closed
• Support change management for trashing data files from ACID tables.: Closed
• Support bootstrap replication of ACID tables: Closed
• Create/Replicate Allocate write-id event: Closed
• Replicate ACID/MM tables write operations.: Closed
• Disable timeout of transactions opened by replication task at target cluster: Closed
• Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.: Closed
• ptest optimization - Replicate ACID/MM tables write operations.: Closed
• SQL Script changes for creating txn write notification in 3.2.0 files: Closed
• Bootstrap repl dump with concurrent write and drop of ACID table makes target inconsistent.: Closed
• Hive support bootstrap of ACID/MM tables on an existing policy.: Closed
• Replicate Streaming ingestion with transactional batch size as 1.: Closed