Hive
  1. Hive
  2. HIVE-4233

The TGT gotten from class 'CLIService' should be renewed on time

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: HiveServer2
    • Labels:
      None
    • Environment:

      CentOS release 6.3 (Final)

      jdk1.6.0_31

      HiveServer2 0.10.0-cdh4.2.0

      Kerberos Security

      Description

      When the HIveServer2 have started more than 7 days, I use beeline shell to connect the HiveServer2,all operation failed.

      The log of HiveServer2 shows it was caused by the Kerberos auth failure,the exception stack trace is:

      2013-03-26 11:55:20,932 ERROR hive.ql.metadata.Hive: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1084)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151)
      at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2275)
      at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:358)
      at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:127)
      at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073)
      at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082)
      ... 16 more
      Caused by: java.lang.IllegalStateException: This ticket is no longer valid
      at javax.security.auth.kerberos.KerberosTicket.toString(KerberosTicket.java:601)
      at java.lang.String.valueOf(String.java:2826)
      at java.lang.StringBuilder.append(StringBuilder.java:115)
      at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:120)
      at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:41)
      at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:130)
      at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:328)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:325)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:128)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
      at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
      at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:163)
      ... 20 more

      I check the code of HiveAuthFactory.loginFromKeytab,it does not schedule a timer to renew the TGT. So I suspect this is the reason of the kerberos auth failure?

      Thanks.

      1. 0001-FIX-HIVE-4233.patch
        5 kB
        Dongyong Wang
      2. HIVE-4233.4.patch
        7 kB
        Thejas M Nair
      3. HIVE-4233.5.patch
        7 kB
        Thejas M Nair
      4. HIVE-4233-2.patch
        6 kB
        Thejas M Nair
      5. HIVE-4233-3.patch
        6 kB
        Thejas M Nair

        Activity

        Hide
        Dongyong Wang added a comment -

        Add HiveKerberosReloginHelper to schdule the command to renew the tgt.

        I tested this patch with a kerberos principal whose maxlife is 15 minutes,it does not fail after 15 mintues.

        When doesn't apply this path,the Keberos auth failure always thrown after 15 mintues,and the beeline can't reconnect the HiveServer2.

        Please review this patch ,maybe it can solve this problem.

        Thanks.

        Show
        Dongyong Wang added a comment - Add HiveKerberosReloginHelper to schdule the command to renew the tgt. I tested this patch with a kerberos principal whose maxlife is 15 minutes,it does not fail after 15 mintues. When doesn't apply this path,the Keberos auth failure always thrown after 15 mintues,and the beeline can't reconnect the HiveServer2. Please review this patch ,maybe it can solve this problem. Thanks.
        Hide
        Jitendra Nath Pandey added a comment -

        1. You could use UserGroupInformation#checkTGTAndReloginFromKeytab method that checks the tgt refresh time before relogin, instead of explicitly checking tgt in your code.

        2. Instead of adding a new relogin thread, you could also consider following two options, which are relatively simpler.
        a) call checkTGTAndReloginFromKeytab before every connection,
        b) catch connection failure and call reloginFromKeytab, if you are able to catch this particular failure.
        Hadoop rpc uses (b)

        3. Apache guidelines insist on not putting username in the code as in the javadoc for HiveKerberosReloginHelper.

        Show
        Jitendra Nath Pandey added a comment - 1. You could use UserGroupInformation#checkTGTAndReloginFromKeytab method that checks the tgt refresh time before relogin, instead of explicitly checking tgt in your code. 2. Instead of adding a new relogin thread, you could also consider following two options, which are relatively simpler. a) call checkTGTAndReloginFromKeytab before every connection, b) catch connection failure and call reloginFromKeytab, if you are able to catch this particular failure. Hadoop rpc uses (b) 3. Apache guidelines insist on not putting username in the code as in the javadoc for HiveKerberosReloginHelper.
        Hide
        Thejas M Nair added a comment -

        UserGroupInformation.reloginFromKeytab is not a method available in hadoop 0.20.x, this will break hive build against it. That call should go through the shim layer for hadoop. (similar to the way shim is used for - ShimLoader.getHadoopShims().loginUserFromKeytab).

        Show
        Thejas M Nair added a comment - UserGroupInformation.reloginFromKeytab is not a method available in hadoop 0.20.x, this will break hive build against it. That call should go through the shim layer for hadoop. (similar to the way shim is used for - ShimLoader.getHadoopShims().loginUserFromKeytab).
        Hide
        Thejas M Nair added a comment -

        Hadoop api's already do relogin if ticket has expired. If we add similar logic to metastore client, we can fix this without having an additional thread. (and fix it for other potential metastore users as well). Since the kerberos expiry is usually in hours or days, having a simple logic of doing relogin if the previous login was several minutes back should fix this issue.
        Dongyong Wang Will you be create a new patch for this ? If not, I should be able to upload a new one soon.

        Show
        Thejas M Nair added a comment - Hadoop api's already do relogin if ticket has expired. If we add similar logic to metastore client, we can fix this without having an additional thread. (and fix it for other potential metastore users as well). Since the kerberos expiry is usually in hours or days, having a simple logic of doing relogin if the previous login was several minutes back should fix this issue. Dongyong Wang Will you be create a new patch for this ? If not, I should be able to upload a new one soon.
        Hide
        Jitendra Nath Pandey added a comment -

        Since the kerberos expiry is usually in hours or days, having a simple logic of doing relogin if the previous login was several minutes back should fix this issue.

        That sounds right, UGI.reloginFromKeytab already does it, checks if sufficient time has elapsed. The 'sufficient time' can be configured. This method also checks for TGT refresh time, therefore just calling reloginFromKeytab before every connection should be a reasonable fix.

        Show
        Jitendra Nath Pandey added a comment - Since the kerberos expiry is usually in hours or days, having a simple logic of doing relogin if the previous login was several minutes back should fix this issue. That sounds right, UGI.reloginFromKeytab already does it, checks if sufficient time has elapsed. The 'sufficient time' can be configured. This method also checks for TGT refresh time, therefore just calling reloginFromKeytab before every connection should be a reasonable fix.
        Hide
        Dongyong Wang added a comment -

        Thanks to Thejas and Jitendra's reply. I agree with your solution, my patch is too complex, waiting for your new patch.

        Show
        Dongyong Wang added a comment - Thanks to Thejas and Jitendra's reply. I agree with your solution, my patch is too complex, waiting for your new patch.
        Hide
        Thejas M Nair added a comment -

        HIVE-4233-2.patch - has the proposed changes. But I need to do some more testing.

        btw, if you have a build without the patch, you can work around the issue by starting hive server with embedded metastore. ie

        bin/hiveserver2 -hiveconf hive.metastore.uris=' '
        

        (The hortonworks distribution (HDP 1.2) configured HS2 this way(metastore in local mode), I think that is the reason why I didn't see this in our testing or among our users.)

        Show
        Thejas M Nair added a comment - HIVE-4233 -2.patch - has the proposed changes. But I need to do some more testing. btw, if you have a build without the patch, you can work around the issue by starting hive server with embedded metastore. ie bin/hiveserver2 -hiveconf hive.metastore.uris=' ' (The hortonworks distribution (HDP 1.2) configured HS2 this way(metastore in local mode), I think that is the reason why I didn't see this in our testing or among our users.)
        Hide
        Dongyong Wang added a comment -

        Thanks Thejas,I will try your work around lately. But we use Cloudera Manager to config the HiveServer2 and Metastore Server,I have to find the correct way to config the embeded metastore.

        Show
        Dongyong Wang added a comment - Thanks Thejas,I will try your work around lately. But we use Cloudera Manager to config the HiveServer2 and Metastore Server,I have to find the correct way to config the embeded metastore.
        Hide
        Thejas M Nair added a comment -

        HIVE-4233-3.patch - new patch with some formatting/comments changes.

        Patch is ready for review.
        Reviewboard link - https://reviews.apache.org/r/11046/

        Show
        Thejas M Nair added a comment - HIVE-4233 -3.patch - new patch with some formatting/comments changes. Patch is ready for review. Reviewboard link - https://reviews.apache.org/r/11046/
        Hide
        Thejas M Nair added a comment -

        I did some manual testing with the patch. I saw a different error from what was reported in the jira, but the patch fixed the issue.
        I changed max life of kerberos principal used by hs2 -

        kadmin.local:  modprinc -maxlife "15 min" hive/<host>@EXAMPLE.COM
        
        kadmin.local:  getprinc hive/<host>@EXAMPLE.COM
        Principal: hive/<host>@EXAMPLE.COM
        ...
        Maximum ticket life: 0 days 00:15:00
        Maximum renewable life: 0 days 00:00:00
        ...
        

        The exception I got when (re)connecting after 15 mins. (without the fix).
        org.apache.hive.service.cli.HiveSQLException: Error connect metastore to setup impersonation
        at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:324)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:153)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:116)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2424)
        at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:319)
        ... 11 more
        Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2286)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2297)
        at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2421)
        ... 12 more
        Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
        ... 17 more
        Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:283)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2286)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2297)
        at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2421)
        at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:319)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:153)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:116)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        )
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:329)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164)
        ... 22 more

        
        
        Show
        Thejas M Nair added a comment - I did some manual testing with the patch. I saw a different error from what was reported in the jira, but the patch fixed the issue. I changed max life of kerberos principal used by hs2 - kadmin.local: modprinc -maxlife "15 min" hive/<host>@EXAMPLE.COM kadmin.local: getprinc hive/<host>@EXAMPLE.COM Principal: hive/<host>@EXAMPLE.COM ... Maximum ticket life: 0 days 00:15:00 Maximum renewable life: 0 days 00:00:00 ... The exception I got when (re)connecting after 15 mins. (without the fix). org.apache.hive.service.cli.HiveSQLException: Error connect metastore to setup impersonation at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:324) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:153) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:116) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2424) at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:319) ... 11 more Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2286) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2297) at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2421) ... 12 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) ... 17 more Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:283) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2286) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2297) at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2421) at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:319) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:153) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:116) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:329) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) ... 22 more
        Hide
        Thejas M Nair added a comment -

        Dongyong Wang Thanks for reporting the issue and creating the initial patch!
        Please let me know if you get to try the patch in your cluster.

        Show
        Thejas M Nair added a comment - Dongyong Wang Thanks for reporting the issue and creating the initial patch! Please let me know if you get to try the patch in your cluster.
        Hide
        Dongyong Wang added a comment -

        We deploy the initial patch from 2013-04-15 and the TGT exception does not occur again.
        I just used the beeline client to connect to the HiveServer2 to execute the query can be successful.

        Show
        Dongyong Wang added a comment - We deploy the initial patch from 2013-04-15 and the TGT exception does not occur again. I just used the beeline client to connect to the HiveServer2 to execute the query can be successful.
        Hide
        Dongyong Wang added a comment -

        Sorry,I haven't tried you patch.

        Show
        Dongyong Wang added a comment - Sorry,I haven't tried you patch.
        Hide
        Thejas M Nair added a comment -

        I found an issue with hadoop 0.20, it is not checking if hadoop version is security enabled. Will run more tests with hadoop 0.20 and submit new patch.

        Show
        Thejas M Nair added a comment - I found an issue with hadoop 0.20, it is not checking if hadoop version is security enabled. Will run more tests with hadoop 0.20 and submit new patch.
        Hide
        Prasad Mujumdar added a comment -

        Thejas M Nair are you planning to update the patch for the hadoop 0.20 issue ? If you are busy with other things, I can make the changes.

        Show
        Prasad Mujumdar added a comment - Thejas M Nair are you planning to update the patch for the hadoop 0.20 issue ? If you are busy with other things, I can make the changes.
        Hide
        Thejas M Nair added a comment -

        Prasad Mujumdar Thanks for reminding me about this issue. Attaching new patch with check for security being enabled.

        Show
        Thejas M Nair added a comment - Prasad Mujumdar Thanks for reminding me about this issue. Attaching new patch with check for security being enabled.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12595727/HIVE-4233.4.patch

        ERROR: -1 due to 1 failed/errored test(s), 2758 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/298/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/298/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12595727/HIVE-4233.4.patch ERROR: -1 due to 1 failed/errored test(s), 2758 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/298/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/298/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Thejas M Nair added a comment -

        TestMinimrCliDriver.testCliDriver_parallel_orderby failure is probably because of some flakiness with the test case.
        It passed when ran it again on my machine.

        Show
        Thejas M Nair added a comment - TestMinimrCliDriver.testCliDriver_parallel_orderby failure is probably because of some flakiness with the test case. It passed when ran it again on my machine.
        Hide
        Gunther Hagleitner added a comment -

        Looks mostly good to me. Couple of comments on RB. Thejas M Nair/Prasad Mujumdar is there really no way we can add automated tests for these scenarios?

        Show
        Gunther Hagleitner added a comment - Looks mostly good to me. Couple of comments on RB. Thejas M Nair / Prasad Mujumdar is there really no way we can add automated tests for these scenarios?
        Hide
        Prasad Mujumdar added a comment -

        Gunther Hagleitner Thanks for the review.
        AFAIK, there's no testing framework for kerberos related changes. I am not sure if Knox or other hadoop ecosystem projects has something that we can reuse.

        Show
        Prasad Mujumdar added a comment - Gunther Hagleitner Thanks for the review. AFAIK, there's no testing framework for kerberos related changes. I am not sure if Knox or other hadoop ecosystem projects has something that we can reuse.
        Hide
        Thejas M Nair added a comment -

        HIVE-4233.5.patch new patch addressing review comments.

        Show
        Thejas M Nair added a comment - HIVE-4233 .5.patch new patch addressing review comments.
        Hide
        Gunther Hagleitner added a comment -

        LGTM +1. Will commit if tests pass.

        Show
        Gunther Hagleitner added a comment - LGTM +1. Will commit if tests pass.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12596469/HIVE-4233.5.patch

        ERROR: -1 due to 1 failed/errored test(s), 2767 tests executed
        Failed tests:

        org.apache.hcatalog.pig.TestHCatStorerMulti.testStorePartitionedTable
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/327/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/327/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12596469/HIVE-4233.5.patch ERROR: -1 due to 1 failed/errored test(s), 2767 tests executed Failed tests: org.apache.hcatalog.pig.TestHCatStorerMulti.testStorePartitionedTable Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/327/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/327/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Gunther Hagleitner added a comment -

        The test failure is unrelated. Tests look good for this patch.

        Show
        Gunther Hagleitner added a comment - The test failure is unrelated. Tests look good for this patch.
        Hide
        Gunther Hagleitner added a comment -

        Committed to trunk. Thanks Thejas!

        Show
        Gunther Hagleitner added a comment - Committed to trunk. Thanks Thejas!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2 #341 (See https://builds.apache.org/job/Hive-trunk-hadoop2/341/)
        HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574)

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
        • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
        • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #341 (See https://builds.apache.org/job/Hive-trunk-hadoop2/341/ ) HIVE-4233 : The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574 ) /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2252 (See https://builds.apache.org/job/Hive-trunk-h0.21/2252/)
        HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574)

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
        • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
        • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2252 (See https://builds.apache.org/job/Hive-trunk-h0.21/2252/ ) HIVE-4233 : The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574 ) /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/)
        HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574)

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
        • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
        • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/ ) HIVE-4233 : The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574 ) /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/)
        HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574)

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
        • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
        • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/ ) HIVE-4233 : The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511574 ) /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Dongyong Wang
          • Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development