Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2655

Catalog throws NPE on kerberos re-logins -Possible race bug when Impala catalog does multiple failed kerberos re-logins to metastore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Workaround
    • Affects Version/s: Impala 2.2, Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Catalog

      Description

      Catalog throws NPE on kerberos re-logins -Possible race bug when Impala catalog does multiple failed kerberos re-logins to metastore

      W1013 04:37:00.439453 29190 Client.java:671] Couldn't setup connection for impala/prod1204.kdc.com@prod.com to prod1202.kdc.com/10.24.4.1:8020
      Java exception follows:
      java.lang.NullPointerException
      at javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
      at javax.security.auth.kerberos.KerberosTicket.isCurrent(KerberosTicket.java:518)
      at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:141)
      at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:59)
      at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:155)
      at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:346)
      at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:344)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:343)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
      at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
      at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
      at org.apache.hadoop.ipc.Client.call(Client.java:1438)
      at org.apache.hadoop.ipc.Client.call(Client.java:1399)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
      at com.sun.proxy.$Proxy14.listCachePools(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.listCachePools(ClientNamenodeProtocolTranslatorPB.java:1242)
      at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy15.listCachePools(Unknown Source)
      at org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:46)
      at org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:31)
      at org.apache.hadoop.fs.BatchedRemoteIterator.makeRequest(BatchedRemoteIterator.java:77)
      at org.apache.hadoop.fs.BatchedRemoteIterator.makeRequestIfNeeded(BatchedRemoteIterator.java:85)
      at org.apache.hadoop.fs.BatchedRemoteIterator.hasNext(BatchedRemoteIterator.java:99)
      at com.cloudera.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:151)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mala_ck Mala Chikka Kempanna
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: