[STORM-180] Nimbus fails to start In case of non-worked zookeeper server specifed in config

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: storm-core
• Labels: nimbus zookeeper
• Environment: Storm 0.9.0.1
• Assignee: Unassigned
• Reporter: Alexander Yerenkow
• Created: 20/Dec/13 12:32
• Updated: 09/Oct/15 00:53
• Resolved: 23/Dec/13 15:59

Description

For example, If I specify three servers in config for zookeeper, and any of them not working during
`storm nimbus`
then nimbus fails to start at all -

Can this be avoided somehow?
If one of zookeeper servers failed, then I had to reconfigure all storm instances to simply start it and connect to working zk servers.

Backtrace:

2013-12-20 14:08:08 b.s.d.nimbus [ERROR] Error on initialization of server service-handler
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:28) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.zookeeper$exists_node_QMARK_$fn__991.invoke(zookeeper.clj:82) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:78) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:92) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.cluster$mk_distributed_cluster_state.invoke(cluster.clj:26) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.cluster$mk_storm_cluster_state.invoke(cluster.clj:201) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:51) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.daemon.nimbus$fn_5528$exec_fn1229auto___5529.invoke(nimbus.clj:884) ~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.applyToHelper(AFn.java:163) ~[clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
at backtype.storm.daemon.nimbus$fn_5528$service_handler_5616.doInvoke(nimbus.clj:881) ~[storm-core-0.9.0.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.4.0.jar:na]
at backtype.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:1136) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.daemon.nimbus$_launch.invoke(nimbus.clj:1167) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.daemon.nimbus$_main.invoke(nimbus.clj:1189) ~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.applyToHelper(AFn.java:159) ~[clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
at backtype.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-0.9.0.1.jar:na]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:72) ~[curator-client-1.0.1.jar:na]
at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:74) ~[curator-client-1.0.1.jar:na]
at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:353) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:149) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:138) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:85) ~[curator-client-1.0.1.jar:na]
at com.netflix.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:134) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:125) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:34) ~[curator-framework-1.0.1.jar:na]
at backtype.storm.zookeeper$exists_node_QMARK_$fn__991.invoke(zookeeper.clj:81) ~[storm-core-0.9.0.1.jar:na]
... 17 common frames omitted
2013-12-20 14:08:08 b.s.util [INFO] Halting process: ("Error on initialization")