[IMPALA-2655] Catalog throws NPE on kerberos re-logins -Possible race bug when Impala catalog does multiple failed kerberos re-logins to metastore

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Workaround
• Affects Version/s: Impala 2.2, Impala 2.3.0
• Component/s: Catalog
• Labels: catalog-server security
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Mala Chikka Kempanna
• Created: 09/Nov/15 19:36
• Updated: 17/Jan/18 22:11
• Resolved: 26/Sep/17 14:16

Description

Catalog throws NPE on kerberos re-logins -Possible race bug when Impala catalog does multiple failed kerberos re-logins to metastore

W1013 04:37:00.439453 29190 Client.java:671] Couldn't setup connection for impala/[e-mail] to prod1202.kdc.com/10.24.4.1:8020
Java exception follows:
java.lang.NullPointerException
at javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
at javax.security.auth.kerberos.KerberosTicket.isCurrent(KerberosTicket.java:518)
at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:141)
at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:59)
at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:155)
at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:346)
at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:343)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy14.listCachePools(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.listCachePools(ClientNamenodeProtocolTranslatorPB.java:1242)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.listCachePools(Unknown Source)
at org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:46)
at org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:31)
at org.apache.hadoop.fs.BatchedRemoteIterator.makeRequest(BatchedRemoteIterator.java:77)
at org.apache.hadoop.fs.BatchedRemoteIterator.makeRequestIfNeeded(BatchedRemoteIterator.java:85)
at org.apache.hadoop.fs.BatchedRemoteIterator.hasNext(BatchedRemoteIterator.java:99)
at com.cloudera.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• Is contained by: IMPALA-5129 Use Kudu's Kinit code to avoid expensive fork