[HIVE-18781] Create/Replicate Open, Commit (without writes) and Abort Txn events - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

EVENT_OPEN_TXN:
Source Warehouse:

Create new event type EVENT_OPEN_TXN with related message format etc.
When any transaction is opened either by auto-commit mode or multi-statement mode, need to capture this event.
Repl dump should read this event from EventNotificationTable and dump the message.

Target Warehouse:

Repl load should read the event from the dump and get the message.
Open a txn in target warehouse.
Create a map of source txn ID against target txn ID and persist the same in metastore. There should be one map per replication policy (DBName.* incase of DB level replication, DBName.TableName incase of table level replication)

EVENT_COMMIT_TXN (Without writes)

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within the txn.

Source warehouse:

Target warehouse:

Repl load should read this event from the dump.
Validate the source txn ID from the event using the Source-Target Txn ID map maintained in target metastore. Also, need to check if corresponding target txn ID is valid.
If valid, then apply the event and commit the corresponding target transaction.
This new event should be idempotent such that if it is applied twice, then second time it should be loop.

EVENT_ABORT_TXN
Source Warehouse:

Create new event type EVENT_ABORT_TXN with related message format etc.
Capture this event when abort the txn.
Repl dump should read this event from EventNotificationTable and dump the message.

Target Warehouse:

Repl load should read the event from the dump and get the message.
Validate if source txn ID from the event is there in the source-target txn ID map. If not there, just noop the event.
If valid, then Abort the corresponding target txn and remove the entry from source-target txn ID map.

All these new events should be idempotent such that if it is applied twice, then second time it should be noop.