[HIVE-12389] CompactionTxnHandler.cleanEmptyAbortedTxns() should safeguard against huge IN clauses

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.0
• Component/s: Metastore, Transactions
• Target Version/s: 1.3.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 12/Nov/15 02:45
• Updated: 23/Nov/15 17:31
• Resolved: 23/Nov/15 17:31

Description

in extreme situations, due to misconfigurations, it may be possible to have 100Ks or even 1Ms of aborted txns.
This causes delete from TXNS where txn_id in (...) to have a huge IN clause and DB chokes.

Should use something like TxnHandler.TIMED_OUT_TXN_ABORT_BATCH_SIZE to break up delete into multiple queries. (Incidentally the batch size should likely be 1000, not 100, maybe even configurable).

On MySQL for example, it can cause query to fail with

Packet for query is too large (9288598 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable.

Issue Links

• is blocked by: HIVE-11948 Investigate TxnHandler and CompactionTxnHandler to see where we improve concurrency