[HIVE-18192] Introduce WriteID per table rather than using global transaction ID

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: HiveServer2, Transactions
• Labels: ACID DR pull-request-available
• Assignee: Sankar Hariappan
• Reporter: Anishek Agarwal
• Created: 01/Dec/17 06:21
• Updated: 01/Oct/19 22:07
• Resolved: 23/Feb/18 16:32

Description

To support ACID replication, we will be introducing a per table write Id which will replace the transaction id in the primary key for each row in a ACID table.

The current primary key is determined via
<original transaction id, bucketid , rowid>

which will move to
<original write id, bucketid, rowid>

For each table modified by the given transaction will have a table level write ID allocated and a persisted map of global txn id -> to table -> write id for that table has to be maintained to allow Snapshot isolation.

Readers should use the combination of ValidTxnList and ValidWriteIdList(Table) for snapshot isolation.

 

 Hive Replication - ACID Tables.pdf has a section "Per Table Sequences (Write-Id)" with more detials

Issue Links

• is depended upon by: HIVE-18193 Migrate existing ACID tables to use write id per table rather than global transaction id
• is related to: HIVE-18444 when creating transactional table make sure location has no data
• relates to: HIVE-18748 Rename table impacts the ACID behavior as table names are not updated in meta-tables.
• relates to: HIVE-18750 Exchange partition should be disabled on ACID/Insert-only tables with per table write ID.
• relates to: HIVE-18751 ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.
• relates to: HIVE-18753 Correct methods and variables names which uses writeId instead of transactionId.
• relates to: HIVE-18824 ValidWriteIdList config should be defined on tables which has to collect stats after insert
• relates to: HIVE-18864 ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
• relates to: HIVE-18747 Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
• relates to: HIVE-18749 Need to replace transactionId with writeId in RecordIdentifier and other relevant contexts.
• relates to: HIVE-18752 HiveEndPoint, HiveStreamingConnection: Optimise metastore calls to open transactions batch and allocate write Ids.
• relates to: HIVE-18923 ValidWriteIdList snapshot per table can be cached for multi-statement transactions.
• links to: GitHub Pull Request #290