Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-64

Queries fail on secure environment due to an impalad's Kerberos ticket expiring

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 0.6
    • Fix Version/s: Impala 0.6
    • Component/s: None
    • Labels:
      None

      Description

      Queries occasionally fail targeting impalad when running against a secure environment. This seems to be because the instance is unable to talk to other impalads in the cluster due to its ticket expiring. These are short running queries (should complete in < 10 seconds)

      Tuple(id=0 size=36 slots=[Slot(id=0 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=2 type=INT col=2 offset=8 null=(offset=0 mask=2)), Slot(id=4 type=INT col=4 offset=12 null=(offset=0 mask=4)), Slot(id=6 type=INT col=7 offset=16 null=(offset=0 mask=8)), Slot(id=10 type=INT col=10 offset=20 null=(offset=0 mask=10)), Slot(id=11 type=FLOAT col=12 offset=24 null=(offset=0 mask=20)), Slot(id=12 type=FLOAT col=19 offset=28 null=(offset=0 mask=40)), Slot(id=13 type=FLOAT col=13 offset=32 null=(offset=0 mask=80))])
      Tuple(id=1 size=12 slots=[Slot(id=1 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=17 type=INT col=6 offset=8 null=(offset=0 mask=2))])
      Tuple(id=2 size=24 slots=[Slot(id=3 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=8 type=STRING col=1 offset=8 null=(offset=0 mask=2))])
      Tuple(id=3 size=56 slots=[Slot(id=5 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=14 type=STRING col=1 offset=8 null=(offset=0 mask=2)), Slot(id=15 type=STRING col=2 offset=24 null=(offset=0 mask=4)), Slot(id=16 type=STRING col=3 offset=40 null=(offset=0 mask=8))])
      Tuple(id=4 size=24 slots=[Slot(id=7 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=9 type=STRING col=24 offset=8 null=(offset=0 mask=2))])
      Tuple(id=5 size=104 slots=[Slot(id=18 type=STRING col=-1 offset=72 null=(offset=0 mask=10)), Slot(id=19 type=STRING col=-1 offset=88 null=(offset=0 mask=20)), Slot(id=20 type=BIGINT col=-1 offset=8 null=(offset=0 mask=1)), Slot(id=21 type=BIGINT col=-1 offset=16 null=(offset=0 mask=0)), Slot(id=22 type=DOUBLE col=-1 offset=24 null=(offset=0 mask=2)), Slot(id=23 type=BIGINT col=-1 offset=32 null=(offset=0 mask=0)), Slot(id=24 type=DOUBLE col=-1 offset=40 null=(offset=0 mask=4)), Slot(id=25 type=BIGINT col=-1 offset=48 null=(offset=0 mask=0)), Slot(id=26 type=DOUBLE col=-1 offset=56 null=(offset=0 mask=8)), Slot(id=27 type=BIGINT col=-1 offset=64 null=(offset=0 mask=0))])
      E0214 10:00:07.121000 24251 authorization.cc:72] Kerberos: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)
      I0214 10:00:07.159731 24251 status.cc:40] Couldn't open transport for 10.20.80.123:22000(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired))
          @           0x852262  impala::Status::Status()
          @           0x818315  impala::ThriftClientImpl::Open()
          @           0x7b6284  impala::BackendClientCache::GetClient()
          @           0x9b5cca  impala::DataStreamSender::Channel::Init()
          @           0x9b7cbf  impala::DataStreamSender::Init()
          @           0x7fa8e5  impala::PlanFragmentExecutor::Prepare()
          @           0x68bf4b  impala::ImpalaServer::FragmentExecState::Prepare()
          @           0x69c89f  impala::ImpalaServer::StartPlanFragmentExecution()
          @           0x69d876  impala::ImpalaServer::ExecPlanFragment()
          @           0x858cb0  impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment()
          @           0x852f09  impala::ImpalaInternalServiceProcessor::dispatchCall()
          @           0x69e73b  apache::thrift::TDispatchProcessor::process()
          @          0x12c977a  apache::thrift::server::TThreadedServer::Task::run()
          @          0x12cbe72  apache::thrift::concurrency::PthreadThread::threadMain()
          @     0x7ffd037bf7b6  start_thread
          @     0x7ffd02d829cd  clone
      

      I also have seen cause failures when communicating with the Hive Meta Store Service in secure mode. It appears Impalad is not kinit'ing frequently enough:

      13/02/20 11:52:42 INFO hive.metastore: Trying to connect to metastore with URI thrift://impala-centos57-3.ent.cloudera.com:9083
      13/02/20 11:52:42 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:KERBEROS) from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
      13/02/20 11:52:42 DEBUG transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslClientTransport@8dc1f04
      13/02/20 11:52:42 ERROR transport.TSaslTransport: SASL negotiation failure
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
      	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
      	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
      	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:163)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
      	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1114)
      	at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1103)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2206)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:334)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:706)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      	... 40 more
      

      Version: impalad version 0.6 RELEASE (build ac205b9d3c02cbc82f306d68fa7633790fb7a6ad)

        Attachments

          Activity

            People

            • Assignee:
              lskuff Lenni Kuff
              Reporter:
              lskuff Lenni Kuff
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: