Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11616

DelegationTokenSecretManager reuses the same objectstore, which has concurrency issues

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: Metastore
    • Labels:
      None
    • Flags:
      Patch

      Description

      sometime in metastore log, will get below exception, after analysis, we found that :
      when hivemetastore start, the DelegationTokenSecretManager will maintain the same objectstore, see here

      saslServer.startDelegationTokenSecretManager(conf, *baseHandler.getMS()*, ServerMode.METASTORE);
      

      this lead to the cocurrent issue.

      2015-08-18 20:59:10,520 | ERROR | pool-6-thread-200 | Error occurred during processing of message. | org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:296)
      org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
      	at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:154)
      	at org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:88)
      	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:112)
      	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:565)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:596)
      	at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
      	at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
      	at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
      	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
      	at org.apache.thrift.transport.HiveTSaslServerTransport.open(HiveTSaslServerTransport.java:133)
      	at org.apache.thrift.transport.HiveTSaslServerTransport$Factory.getTransport(HiveTSaslServerTransport.java:261)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1652)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
      	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
      	at org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
      	at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
      	at org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
      	at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
      	at org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:420)
      	at org.apache.hadoop.hive.metastore.ObjectStore.getToken(ObjectStore.java:6455)
      	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
      	at com.sun.proxy.$Proxy4.getToken(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:146)
      	... 21 more
      

        Attachments

        1. HIVE-11616.patch
          2 kB
          Cody Fu
        2. HIVE-11616.02.patch
          6 kB
          Sergey Shelukhin
        3. HIVE-11616.01.patch
          5 kB
          Sergey Shelukhin

          Issue Links

            Activity

              People

              • Assignee:
                sershe Sergey Shelukhin
                Reporter:
                wenli wangwenli
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 12h
                  12h
                  Remaining:
                  Remaining Estimate - 12h
                  12h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified