[HIVE-19927] Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: HiveServer2, repl, Transactions
• Labels: DR pull-request-available replication
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 18/Jun/18 00:45
• Updated: 17/Nov/22 08:54
• Resolved: 28/Jul/18 05:46

Description

During bootstrap dump of ACID tables, let's consider the below sequence.

• Current session (REPL DUMP), Open txn (Txn1) - Event-10
• Another session (Session-2), Open txn (Txn2) - Event-11
• Session-2 -> Insert data (T1.D1) to ACID table. - Event-12
• Get lastReplId = last event ID logged. (Event-12)
• Session-2 -> Commit Txn (Txn2) - Event-13
• Dump ACID tables based on validTxnList based on Txn1. --> This step skips all the data written by txns > Txn1. So, T1.D1 will be missing.
• Commit Txn (Txn1)
• REPL LOAD from bootstrap dump will skip T1.D1.
• Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is opened after Txn1. So, data T1.D1 will be lost for ever.

Proposed to capture the lastReplId of bootstrap before opening current txn (Txn1) and store it in Driver context and use it for dump.

Issue Links

• links to: GitHub Pull Request #403