[AMBARI-17258] Storm service check failed after Ambari upgrade - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: None
Component/s: ambari-server
Labels:
None

Description

Marked this as blocker since the issue affected many EU End to End test runs that involved running a service check as part of pre-check condition for EU

ambari-server-2.4.0.0-665.x86_64
ambari-server --hash

Steps

Deploy HDP-2.3.2.0 cluster with Ambari 2.1.2 (secure, non-HA cluster)
Upgrade Ambari to 2.4.0.0-665
Stop and Start all services
Run service check for Storm

Result
Failure as below:

stderr:   /var/lib/ambari-agent/data/errors-729.txt

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 79, in <module>
ServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 257, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 70, in service_check
user=params.storm_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'storm jar /tmp/wordCount.jar storm.starter.WordCountTopology WordCountid16ac396c_date241516' returned 1. 951  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
1088 [main] INFO  b.s.u.Utils - Using storm.yaml from resources
1184 [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
1240 [main] INFO  b.s.u.Utils - Using storm.yaml from resources
1285 [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7463484230184273671:-7128399429695564111
1287 [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
1292 [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1352 [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]
1615 [main] INFO  o.a.s.z.Login - successfully logged in.
1650 [main] ERROR o.a.t.t.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[?:1.7.0_67]
at org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115) ~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454) ~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115) ~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454) ~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
1666 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Client failed to open SaslClientTransport to interact with a server during session initiation: org.apache.thrift7.transport.TTransportException: GSS initiate failed
org.apache.thrift7.transport.TTransportException: GSS initiate failed
at org.apache.thrift7.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:316) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
1675 [main] WARN  b.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from localhost. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: SASL authentication not complete
at org.apache.thrift7.transport.TSaslTransport.write(TSaslTransport.java:474) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.write(TSaslClientTransport.java:37) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:62) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.send_getClusterInfo(Nimbus.java:553) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:546) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
Exception in thread "main" java.lang.RuntimeException: Could not find leader nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus hosts for config nimbus.seeds
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
stdout:   /var/lib/ambari-agent/data/output-729.txt

2016-06-15 06:24:45,500 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-06-15 06:24:45,503 - checked_call['hostid'] {}
2016-06-15 06:24:45,516 - checked_call returned (0, '16ac396c')
2016-06-15 06:24:45,517 - File['/tmp/wordCount.jar'] {'owner': 'storm', 'content': StaticFile('wordCount.jar')}
2016-06-15 06:24:45,526 - Execute['storm jar /tmp/wordCount.jar storm.starter.WordCountTopology WordCountid16ac396c_date241516'] {'logoutput': True, 'path': ['/usr/hdp/current/storm-client/bin'], 'user': 'storm'}
951  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
1088 [main] INFO  b.s.u.Utils - Using storm.yaml from resources
1184 [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
1240 [main] INFO  b.s.u.Utils - Using storm.yaml from resources
1285 [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7463484230184273671:-7128399429695564111
1287 [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
1292 [main] WARN  b.s.u.NimbusClient - Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
1352 [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]
1615 [main] INFO  o.a.s.z.Login - successfully logged in.
1650 [main] ERROR o.a.t.t.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[?:1.7.0_67]
at org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115) ~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454) ~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309) ~[?:1.7.0_67]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115) ~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454) ~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.7.0_67]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[?:1.7.0_67]
... 18 more
1666 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Client failed to open SaslClientTransport to interact with a server during session initiation: org.apache.thrift7.transport.TTransportException: GSS initiate failed
org.apache.thrift7.transport.TTransportException: GSS initiate failed
at org.apache.thrift7.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:316) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
1675 [main] WARN  b.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from localhost. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: SASL authentication not complete
at org.apache.thrift7.transport.TSaslTransport.write(TSaslTransport.java:474) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslClientTransport.write(TSaslClientTransport.java:37) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:62) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.send_getClusterInfo(Nimbus.java:553) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:546) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [wordCount.jar:?]
Exception in thread "main" java.lang.RuntimeException: Could not find leader nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus hosts for config nimbus.seeds
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)

Command failed after 1 tries

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

AMBARI-17258.patch
15/Jun/16 16:55
3 kB
Dmitry Lysnichenko

Issue Links

links to

review_request_

Activity

People

Assignee:: Dmitry Lysnichenko

Reporter:: Dmitry Lysnichenko

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 15/Jun/16 16:55

Updated:: 16/Jun/16 13:00

Resolved:: 16/Jun/16 10:05