[HIVE-18781] Create/Replicate Open, Commit (without writes) and Abort Txn events

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: repl, Transactions
• Labels: ACID DR pull-request-available replication
• Target Version/s: 3.0.0
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Due: 26/Feb/18
• Created: 23/Feb/18 04:33
• Updated: 22/May/18 23:15
• Resolved: 02/Apr/18 04:36

Description

EVENT_OPEN_TXN:
Source Warehouse:

• Create new event type EVENT_OPEN_TXN with related message format etc.
• When any transaction is opened either by auto-commit mode or multi-statement mode, need to capture this event.
• Repl dump should read this event from EventNotificationTable and dump the message.

Target Warehouse:

• Repl load should read the event from the dump and get the message.
• Open a txn in target warehouse.
• Create a map of source txn ID against target txn ID and persist the same in metastore. There should be one map per replication policy (DBName.* incase of DB level replication, DBName.TableName incase of table level replication)

 

EVENT_COMMIT_TXN (Without writes)

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within the txn.

Source warehouse:

• Create EVENT_COMMIT_TXN event type with corresponding message format etc.

Target warehouse:

• Repl load should read this event from the dump.
• Validate the source txn ID from the event using the Source-Target Txn ID map maintained in target metastore. Also, need to check if corresponding target txn ID is valid.
• If valid, then apply the event and commit the corresponding target transaction.
• This new event should be idempotent such that if it is applied twice, then second time it should be loop.

 

EVENT_ABORT_TXN
Source Warehouse:

• Create new event type EVENT_ABORT_TXN with related message format etc.
• Capture this event when abort the txn.
• Repl dump should read this event from EventNotificationTable and dump the message.

Target Warehouse:

• Repl load should read the event from the dump and get the message.
• Validate if source txn ID from the event is there in the source-target txn ID map. If not there, just noop the event.
• If valid, then Abort the corresponding target txn and remove the entry from source-target txn ID map.

All these new events should be idempotent such that if it is applied twice, then second time it should be noop.

Issue Links

• is cloned by: HIVE-19089 Create/Replicate Allocate write-id event
• links to: GitHub Pull Request #311