Hive
  1. Hive
  2. HIVE-4233

The TGT gotten from class 'CLIService' should be renewed on time

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: HiveServer2
    • Labels:
      None
    • Environment:

      CentOS release 6.3 (Final)

      jdk1.6.0_31

      HiveServer2 0.10.0-cdh4.2.0

      Kerberos Security

      Description

      When the HIveServer2 have started more than 7 days, I use beeline shell to connect the HiveServer2,all operation failed.

      The log of HiveServer2 shows it was caused by the Kerberos auth failure,the exception stack trace is:

      2013-03-26 11:55:20,932 ERROR hive.ql.metadata.Hive: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1084)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151)
      at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2275)
      at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:358)
      at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:127)
      at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073)
      at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082)
      ... 16 more
      Caused by: java.lang.IllegalStateException: This ticket is no longer valid
      at javax.security.auth.kerberos.KerberosTicket.toString(KerberosTicket.java:601)
      at java.lang.String.valueOf(String.java:2826)
      at java.lang.StringBuilder.append(StringBuilder.java:115)
      at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:120)
      at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:41)
      at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:130)
      at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:328)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:325)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:128)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
      at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
      at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:163)
      ... 20 more

      I check the code of HiveAuthFactory.loginFromKeytab,it does not schedule a timer to renew the TGT. So I suspect this is the reason of the kerberos auth failure?

      Thanks.

      1. 0001-FIX-HIVE-4233.patch
        5 kB
        Dongyong Wang
      2. HIVE-4233.4.patch
        7 kB
        Thejas M Nair
      3. HIVE-4233.5.patch
        7 kB
        Thejas M Nair
      4. HIVE-4233-2.patch
        6 kB
        Thejas M Nair
      5. HIVE-4233-3.patch
        6 kB
        Thejas M Nair

        Activity

        Dongyong Wang created issue -
        Dongyong Wang made changes -
        Field Original Value New Value
        Summary The TGT gotten from class 'CLIService' should be renewed on time? The TGT gotten from class 'CLIService' should be renewed on time
        Dongyong Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Dongyong Wang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Dongyong Wang made changes -
        Attachment 0001-FIX-HIVE-4233.patch [ 12578711 ]
        Dongyong Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Thejas M Nair made changes -
        Attachment HIVE-4233-2.patch [ 12582594 ]
        Thejas M Nair made changes -
        Attachment HIVE-4233-3.patch [ 12582643 ]
        Thejas M Nair made changes -
        Assignee Thejas M Nair [ thejas ]
        Thejas M Nair made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Thejas M Nair made changes -
        Attachment HIVE-4233.4.patch [ 12595727 ]
        Thejas M Nair made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.11.0 [ 12323587 ]
        Affects Version/s 0.10.0 [ 12320745 ]
        Thejas M Nair made changes -
        Attachment HIVE-4233.5.patch [ 12596469 ]
        Thejas M Nair made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Thejas M Nair made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gunther Hagleitner made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.12.0 [ 12324312 ]
        Resolution Fixed [ 1 ]
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Dongyong Wang
          • Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development