Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12389

CompactionTxnHandler.cleanEmptyAbortedTxns() should safeguard against huge IN clauses

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.3.0
    • Metastore, Transactions
    • None

    Description

      in extreme situations, due to misconfigurations, it may be possible to have 100Ks or even 1Ms of aborted txns.
      This causes delete from TXNS where txn_id in (...) to have a huge IN clause and DB chokes.

      Should use something like TxnHandler.TIMED_OUT_TXN_ABORT_BATCH_SIZE to break up delete into multiple queries. (Incidentally the batch size should likely be 1000, not 100, maybe even configurable).

      On MySQL for example, it can cause query to fail with

      Packet for query is too large (9288598 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable.

      Attachments

        1. HIVE-12389.patch
          6 kB
          Eugene Koifman
        2. HIVE-12389.2.patch
          7 kB
          Eugene Koifman
        3. HIVE-12389.3.patch
          7 kB
          Eugene Koifman

        Issue Links

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: