[AMBARI-21129] Nimbus fails to start when Ambari is upgraded to 2.5.1, EU to HDP 2.6.1, and cluster is then Kerberized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: 2.5.2
• Component/s: stacks
• Assignee: Alejandro Fernandez
• Reporter: Denis Tarasyuk
• Created: 25/May/17 18:12
• Updated: 02/Jun/17 23:23
• Resolved: 02/Jun/17 22:48

Description

STR:

1. Ambari 2.5.0 with Storm and HDP 2.6 and Storm
2. Upgrade Ambari to 2.5.1
3. EU to HDP 2.6.1
4. Kerberize the cluster

The exception found in nimbus.log:

2017-05-25 11:31:10.235 o.a.s.n.NimbusInfo [INFO] Nimbus figures out its name to natu146-fpas-dlm30tofnsec-u14-re-re-4.openstacklocal
2017-05-25 11:31:10.247 o.a.s.d.nimbus [ERROR] Error on initialization of server service-handler
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.storm.hdfs.security.AutoHDFS
	at org.apache.storm.security.auth.AuthUtils.getNimbusAutoCredPlugins(AuthUtils.java:188) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:223) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$fn__9066$exec_fn__2832__auto____9067.invoke(nimbus.clj:2416) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
	at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus$fn__9066$service_handler__9099.doInvoke(nimbus.clj:2413) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2501) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2534) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2557) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
Caused by: java.lang.ClassNotFoundException: org.apache.storm.hdfs.security.AutoHDFS
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[?:1.7.0_67]
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[?:1.7.0_67]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[?:1.7.0_67]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[?:1.7.0_67]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[?:1.7.0_67]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[?:1.7.0_67]
	at java.lang.Class.forName0(Native Method) ~[?:1.7.0_67]
	at java.lang.Class.forName(Class.java:190) ~[?:1.7.0_67]
	at org.apache.storm.security.auth.AuthUtils.getNimbusAutoCredPlugins(AuthUtils.java:181) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	... 13 more
2017-05-25 11:31:10.295 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus$fn__9066$service_handler__9099.doInvoke(nimbus.clj:2413) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2501) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2534) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2557) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-1.1.0.2.6.1.0-109.jar:1.1.0.2.6.1.0-109]

This is because upgrading Ambari from an older version to 2.5.1 right now will update storm-env only if Storm is present and the cluster is Kerberized.
In this case, Ambari was upgraded to 2.5.1, EU was performed to HDP 2.6.1, and then the cluster was kerberized; kerberization has no trigger to change configs and will not invoke Stack Advisor.

The fix is to relax the constraint so that Ambari Upgrade adds the exports to storm-env as long as Storm is present. If Storm is added after the fact, then its definition inside of HDP 2.6 (plus Stack Advisor) will handle that use case.