[HIVE-12636] Ensure that all queries (with DbTxnManager) run in a transaction

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 09/Dec/15 20:20
• Updated: 22/May/18 23:59
• Resolved: 01/May/17 16:44

Description

Assuming Hive is using DbTxnManager
Currently (as of this writing only auto commit mode is supported), only queries that write to an Acid table start a transaction.
Read-only queries don't open a txn but still acquire locks.
This makes internal structures confusing/odd.
The are constantly 2 code paths to deal with which is inconvenient and error prone.

Also, a txn id is convenient "handle" for all locks/resources within a txn.
Doing thing would mean the client no longer needs to track locks that it acquired. This enables further improvements to metastore side of Acid.

1. add metastore call to openTxn() and acquireLocks() in a single call. this it to make sure perf doesn't degrade for read-only query. (Would also be useful for auto commit write queries)
2. Should RO queries generate txn ids from the same sequence? (they could for example use negative values of a different sequence). Txnid is part of the delta/base file name. Currently it's 7 digits. If we use the same sequence, we'll exceed 7 digits faster. (possible upgrade issue). On the other hand there is value in being able to pick txn id and commit timestamp out of the same logical sequence.

Issue Links

• blocks: HIVE-16376 Driver.queryState.commandType is often not set correctly
• blocks: HIVE-16377 Clean up the code now that all locks belong to a transaction
• is depended upon by: HIVE-14879 integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones
• is related to: HIVE-15949 Make DbTxnManager.openTxn() lockin the sanpshot
• is related to: HIVE-16447 QueryPlan.getOperation() is not always correct
• relates to: HIVE-16442 ExplainSemanticAnalyzer shares QueryState between statements
• relates to: HIVE-16360 Improve "No delta files or original files found to compact in" message

Sub-Tasks

• enable Acid by default in the parent patch and run build bot: Resolved