Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6409

FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: HCatalog
    • Labels:
      None
    • Tags:
      FileOutputCommitterContainer delegation tokens

      Description

      When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls the underlying OutputCommitter and then attempts to register partitions in HCatalog.

      If the commit fails (for example, because of HIVE-4996), commitJob() cancels delegation tokens retrieved from HCatalog before the exception is rethrown.

      java.io.IOException: java.lang.reflect.InvocationTargetException
          at
      org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
          at
      org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
          at
      org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
          at
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at
      org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
          ... 5 more
      Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
      partition to metastore. Cause :
      MetaException(message:java.lang.RuntimeException: commitTransaction was called
      but openTransactionCalls = 0. This probably indicates that there are unbalanced
      calls to openTransaction/commitTransaction)
          at
      org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
      

      The problem is that this happens before abortJob() has had a chance to run, thus yielding the following error:

      MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password
      	at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
      	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
      	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
      	at java.security.AccessController.doPrivileged(Native Method)
      ...
      

      I'll have a patch out that only cancels delegation tokens if the commitJob() has succeeded.

        Attachments

        1. HIVE-6409.patch
          1 kB
          Mithun Radhakrishnan

          Activity

            People

            • Assignee:
              mithun Mithun Radhakrishnan
              Reporter:
              mithun Mithun Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: