Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24549

TxnManager should not be shared across queries

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • HiveServer2
    • None

    Description

      There are various sections of code that assume the DbTxnManager is not shared across concurrent queries in a session.
      Such as (which gets invoked during closeOperation):
      https://github.com/apache/hive/blob/3f5e01cae5b65dde7edb3fbde8ebe70c1d02f6cf/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L868-L885

         // is usually called after close() to commit or rollback a query and end the driver life cycle.
        // do not understand why it is needed and wonder if it could be combined with close.
        @Override
        public void destroy() {
          driverState.lock();
          try {
            // in the cancel case where the driver state is INTERRUPTED, destroy will be deferred to
            // the query process
            if (driverState.isDestroyed()) {
              return;
            } else {
              driverState.descroyed();
            }
          } finally {
            driverState.unlock();
          }
          driverTxnHandler.destroy();
        }
      

      The problematic part is the: driverTxnHandler.destroy() which looks like:

       void destroy() {
         boolean isTxnOpen =
           driverContext != null &&
           driverContext.getTxnManager() != null &&
           driverContext.getTxnManager().isTxnOpen();
         release(!hiveLocks.isEmpty() || isTxnOpen);
       }
      

      What happens is (rough sketch):
      Q1 - starts operation, acquires txn, does operation, closes txn/cleans up txn info, starts fetching data
      Q2 - starts operation, acquire txn
      Q1 - calls close operation which in turn calls destroy which sees the Q2s transaction information and cleans it up.
      Q2 - proceeds and fails in splitGeneration when it no longer can find its Valid*TxnIdList information.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jfs John Sherman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: