[HIVE-18747] Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Labels: ACID pull-request-available
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 20/Feb/18 08:21
• Updated: 08/Nov/18 01:15
• Resolved: 03/Apr/18 03:27

Description

Per table write ID implementation (HIVE-18192) maintains a map between txn ID and table write ID in TXN_TO_WRITE_ID meta table. 

The entries in this table is used to generate ValidWriteIdList for the given ValidTxnList to ensure snapshot isolation. 

When table or database is dropped, then these entries are cleaned-up. But, it is necessary to clean-up for active tables too for better performance.

TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID.  The state of each Write ID (open, committed, aborted) is determined by the state of the parent transaction.  In order to be able to get a WriteIdList that is accurate wrt ValidTxnList that is locked in at the start of the transaction, we have to retain txnid<->writeid mapping even after the transaction ends. This is because a reader at Snapshot Isolation that started when transaction X was open, should continue to ignore the data written by X even after X commits.

So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID.  There are 2 parts to it. When txn X is opened, it records Y=select min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) table, i.e. it adds (X, Y) to MIN_HISTORY.  On commit (and abort) of X, it removes its own entry from MIN_HISTORY. In the absence of Aborted transactions, MIN_HISTORY gives us the smallest open txnid across all active reader snapshots.  Let Z=select min(opentxnid) from MIN_HISTORY. We can delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since every active reader sees txns < Z as committed.

If S is aborted txns, we retain the metadata about it in TXNS as long as any data written S may be visible to some reader in the system so that the reader knows to skip this data.  The rules for when that is are complex but wrt to TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A).  

If no open or aborted txns exist in the system, then we need to enable cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next).  

Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator immediately after cleaning up aborted txns metadata from TXNS table.

 

Issue Links

• blocks: HIVE-18772 Make Acid Cleaner use MIN_HISTORY_LEVEL
• is related to: HIVE-18192 Introduce WriteID per table rather than using global transaction ID
• links to: GitHub Pull Request #321