Apache S4
  1. Apache S4
  2. S4-107

zkClient connect timeout when deploy application

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6
    • Labels:
      None

      Description

      When use the branch S4-95, commit 995bf22, I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

      From the log I saw that S4 node can acquire task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

       
      -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
      15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
      - comm module class [org.apache.s4.comm.DefaultCommModule]
      - comm configuration file [default.s4.comm.properties from classpath]
      - core module class [org.apache.s4.core.DefaultCoreModule]
      - core configuration file[default.s4.core.properties from classpath]
      - extra modules: []
      - inline parameters: []
      15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
      15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
      15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
      15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
      15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
      15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
      15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
      15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
      15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
      15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
      15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
      com.google.inject.ProvisionException: Guice provision errors:
      
      1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
      while locating org.apache.s4.comm.topology.ClustersFromZK
      while locating org.apache.s4.comm.topology.Clusters
      for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
      while locating org.apache.s4.core.RemoteSenders
      for field at org.apache.s4.core.App.remoteSenders(App.java:49)
      while locating matrix.MatrixApp
      Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
      at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
      at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
      at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
      at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
      at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
      at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
      at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
      at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
      at org.apache.s4.core.Server.loadApp(Server.java:123)
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
      at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
      at org.apache.s4.core.Server.start(Server.java:87)
      at org.apache.s4.core.Main.startNode(Main.java:177)
      at org.apache.s4.core.Main.main(Main.java:86)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
      at org.apache.s4.tools.Tools.main(Tools.java:95)
      
      2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
      at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
      while locating org.apache.s4.comm.topology.RemoteStreams
      for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
      while locating org.apache.s4.core.RemoteSenders
      for field at org.apache.s4.core.App.remoteSenders(App.java:49)
      while locating matrix.MatrixApp
      Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
      at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
      at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
      at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
      at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
      at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
      at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      at com.google.inject.Scopes$1$1.get(Scopes.java:65)
      at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
      at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
      at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
      at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
      at org.apache.s4.core.Server.loadApp(Server.java:123)
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
      at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
      at org.apache.s4.core.Server.start(Server.java:87)
      at org.apache.s4.core.Main.startNode(Main.java:177)
      at org.apache.s4.core.Main.main(Main.java:86)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
      at org.apache.s4.tools.Tools.main(Tools.java:95)
      
      3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
      at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
      while locating org.apache.s4.comm.topology.RemoteStreams
      for field at org.apache.s4.core.App.remoteStreams(App.java:49)
      while locating matrix.MatrixApp
      Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
      at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
      at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
      at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
      at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
      at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
      at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      at com.google.inject.Scopes$1$1.get(Scopes.java:65)
      at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
      at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
      at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
      at org.apache.s4.core.Server.loadApp(Server.java:123)
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
      at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
      at org.apache.s4.core.Server.start(Server.java:87)
      at org.apache.s4.core.Main.startNode(Main.java:177)
      at org.apache.s4.core.Main.main(Main.java:86)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
      at org.apache.s4.tools.Tools.main(Tools.java:95)
      
      3 errors
      at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
      at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
      at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
      15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
      org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
      at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
      at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
      

      Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

       
      -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
      15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
      - comm module class [org.apache.s4.comm.DefaultCommModule]
      - comm configuration file [default.s4.comm.properties from classpath]
      - core module class [org.apache.s4.core.DefaultCoreModule]
      - core configuration file[default.s4.core.properties from classpath]
      - extra modules: []
      - inline parameters: []
      15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
      15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
      15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
      15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
      15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
      15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
      15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
      15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
      15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
      15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
      15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
      15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
      15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
      15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
      15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
      com.google.inject.ProvisionException: Guice provision errors:
      
      1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
      while locating org.apache.s4.comm.topology.ClustersFromZK
      while locating org.apache.s4.comm.topology.Clusters
      for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
      while locating org.apache.s4.core.RemoteSenders
      for field at org.apache.s4.core.App.remoteSenders(App.java:49)
      while locating matrix.MatrixApp
      
      1 error
      at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
      at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
      at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
      at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
      Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
      at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
      at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
      at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
      at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
      at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
      at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
      at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
      at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
      at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
      at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
      at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
      at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
      ... 5 common frames omitted
      1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
      org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
      at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
      at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
      at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
      at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
      

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          97d 16h 44m 1 Matthieu Morel 27/Feb/13 10:27
          Matthieu Morel made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Matthieu Morel made changes -
          Link This issue Is contained by S4-108 [ S4-108 ]
          Hide
          Matthieu Morel added a comment -

          With S4-108 we minimize the connections to Zookeeper

          Show
          Matthieu Morel added a comment - With S4-108 we minimize the connections to Zookeeper
          Matthieu Morel made changes -
          Affects Version/s 0.5.0 [ 12318653 ]
          Fix Version/s 0.6 [ 12321702 ]
          Hide
          Aimee Cheng added a comment -

          Ok, no problem.

          Show
          Aimee Cheng added a comment - Ok, no problem.
          Hide
          Matthieu Morel added a comment -

          Thanks Aimee for the feedback. Current workaround is to change the limit in ZooKeeper configuration. see maxClientCnxns in http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_advancedConfiguration

          I also added a specific ticket for that purpose: S4-108 . It seems that the connection timeout issues are specifically due to this problem, so if you agree, we should mark this current ticket as superseded by S4-108.

          Show
          Matthieu Morel added a comment - Thanks Aimee for the feedback. Current workaround is to change the limit in ZooKeeper configuration. see maxClientCnxns in http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_advancedConfiguration I also added a specific ticket for that purpose: S4-108 . It seems that the connection timeout issues are specifically due to this problem, so if you agree, we should mark this current ticket as superseded by S4-108 .
          Hide
          Aimee Cheng added a comment -

          This problem is caused by too many zkclient connections in 1 node. I checked the log of zookeeper and found that log. So when I start more than 1 node in the same hosts, the connection timeout often happens.

          Fri Nov 23 17:36:31 2012: 2012-11-23 17:36:31,814 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from /98.138.151.53 - max is 10
          

          Maybe need refine the code and combine some zookeeper connections.

          Show
          Aimee Cheng added a comment - This problem is caused by too many zkclient connections in 1 node. I checked the log of zookeeper and found that log. So when I start more than 1 node in the same hosts, the connection timeout often happens. Fri Nov 23 17:36:31 2012: 2012-11-23 17:36:31,814 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from /98.138.151.53 - max is 10 Maybe need refine the code and combine some zookeeper connections.
          Hide
          Aimee Cheng added a comment -

          The zookeeper server version is 3.3.3 and I am sure the zookeeper server is running. "./s4 status" can show information. I think the log also shows that the s4 node can connect the zookeeper server.

           
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          

          Sometimes when I start two nodes in the same host, 1 node can work but another node always fail.

          Show
          Aimee Cheng added a comment - The zookeeper server version is 3.3.3 and I am sure the zookeeper server is running. "./s4 status" can show information. I think the log also shows that the s4 node can connect the zookeeper server. 15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044 Sometimes when I start two nodes in the same host, 1 node can work but another node always fail.
          Hide
          Matthieu Morel added a comment -

          Thanks for reporting that. Unfortunately the message from zkclient is not very helpful.

          Can you make sure you have connectivity? Does "s4 status" work?

          If I remember well, there might also be a problem with zkclient, if you have a 3.4+ Zookeeper server, because of an API change. If that is the case, we'll create a new ticket.

          Show
          Matthieu Morel added a comment - Thanks for reporting that. Unfortunately the message from zkclient is not very helpful. Can you make sure you have connectivity? Does "s4 status" work? If I remember well, there might also be a problem with zkclient, if you have a 3.4+ Zookeeper server, because of an API change. If that is the case, we'll create a new ticket.
          Matthieu Morel made changes -
          Link This issue is related to S4-95 [ S4-95 ]
          Aimee Cheng made changes -
          Description When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          {noformat}


          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I saw that S4 node can acquire task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          {noformat}


          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          Aimee Cheng made changes -
          Description When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          {quote}
          {noformat}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          {noformat}
          {quote}

          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.
          {quote}
          {noformat}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          {quote}
          When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          {noformat}


          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          {noformat}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          Aimee Cheng made changes -
          Description When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]

          {quote}
          {noformat}
          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          {quote}
          When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          {quote}
          {noformat}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          {noformat}
          {quote}

          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.
          {quote}
          {noformat}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          {quote}
          Aimee Cheng made changes -
          Description When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]

          {quote}

          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]

          {quote}
          {noformat}
          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          {noformat}
          {quote}
          Aimee Cheng made changes -
          Field Original Value New Value
          Description When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.

          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]


          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          When use the branch S4-95, commit [995bf22|https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=f9689ea0055e1d7a2a8459b8ee380283767f0ac8], I cannot deploy application successful, when inject RemoteStreams the zkclient will connect timeout.

          From the log I see that S4 node can acquired task successfully at beginning, but when load s4r file it will fail because of zkClient connection timeout. I checked that the zookeeper address passed to zkClient was right. And in another case, in the same hosts, the node which started at first can run well, but another node cannot start with the above exception.
          {quote}
          -bash-4.1$ ./s4 node -c=cluster10 -zk=fed2043:2181
          15:28:52.117 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:28:52.158 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:28:52.832 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:28:53.041 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414046; state is : SyncConnected
          15:28:53.208 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-0 by fed2044
          15:28:53.300 [Main Thread] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1384451834145606764s4r]
          15:28:53.303 [Main Thread] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353511733302-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:28:53.303 [Main Thread] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353511733302-0]
          15:28:53.469 [Main Thread] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:28:53.520 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster10,mode=unicast,type=,nodes=[{partition=2,port=12002,machineName=fed2044,taskId=Task-2}, {partition=0,port=12000,machineName=fed2044,taskId=Task-0}]} from null
          15:28:53.616 [Main Thread] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d34519
          15:29:26.477 [Main Thread] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1384451834145606764s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61)
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          2) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for parameter 1 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:94)
          at org.apache.s4.comm.topology.RemoteStreams.class(RemoteStreams.java:57)
          while locating org.apache.s4.comm.topology.RemoteStreams
          for field at org.apache.s4.core.App.remoteStreams(App.java:49)
          while locating matrix.MatrixApp
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93)
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81)
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49)
          at org.apache.s4.comm.topology.RemoteStreams.<init>(RemoteStreams.java:97)
          at org.apache.s4.comm.topology.RemoteStreams$$FastClassByGuice$$47a2c59.newInstance(<generated>)
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
          at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.inject.Scopes$1$1.get(Scopes.java:65)
          at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76)
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73)
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73)
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60)
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945)
          at org.apache.s4.core.Server.loadApp(Server.java:123)
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128)
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181)
          at org.apache.s4.core.Server.start(Server.java:87)
          at org.apache.s4.core.Main.startNode(Main.java:177)
          at org.apache.s4.core.Main.main(Main.java:86)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54)
          at org.apache.s4.tools.Tools.main(Tools.java:95)

          3 errors
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          15:29:26.480 [Main Thread] ERROR o.a.s.d.DistributedDeploymentManager - Cannot deploy application
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.start(DistributedDeploymentManager.java:181) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Server.start(Server.java:87) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.startNode(Main.java:177) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.core.Main.main(Main.java:86) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_31]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_31]
          at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_31]
          at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:54) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.tools.Tools.main(Tools.java:95) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating]

          {quote}

          Also, if I start a two nodes first, then deploy s4r file, both two nodes will have such problem. Follow is the error log in host fed2044.

          -bash-4.1$ ./s4 node -c=cluster11 -zk=fed2043:2181
          15:45:39.648 [Main Thread] INFO org.apache.s4.core.Main - Initializing S4 node with :
          - comm module class [org.apache.s4.comm.DefaultCommModule]
          - comm configuration file [default.s4.comm.properties from classpath]
          - core module class [org.apache.s4.core.DefaultCoreModule]
          - core configuration file[default.s4.core.properties from classpath]
          - extra modules: []
          - inline parameters: []
          15:45:39.689 [Main Thread] DEBUG org.apache.s4.core.Main - Adding named parameters for injection : [s4.cluster.zk_address=fed2043:2181]
          15:45:40.359 [Main Thread] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
          15:45:40.578 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - New session:88692421596414060; state is : SyncConnected
          15:45:40.771 [Main Thread] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by fed2044
          15:53:03.437 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - Loading application [myApp] from file [/tmp/tmp1117261642880164701s4r]
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] WARN o.a.s4.base.util.S4RLoaderFactory - s4.tmp.dir not specified, using temporary directory [/tmp/1353513183440-0] for unpacking S4R. You may want to specify a parent non-temporary directory.
          15:53:03.441 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.base.util.S4RLoaderFactory - Unzipping S4R archive in [/tmp/1353513183440-0]
          15:53:03.614 [ZkClient-EventThread-16-fed2043:2181] INFO org.apache.s4.core.Server - App class name is: matrix.MatrixApp
          15:53:03.673 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster11,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=fed2044,taskId=Task-0}, {partition=1,port=12001,machineName=fed2044,taskId=Task-1}]} from null
          15:53:03.766 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@16d8b192
          15:53:04.125 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:88692421596414064
          15:53:04.134 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [UserYCT]
          15:53:04.139 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [contentYCT]
          15:53:04.141 [ZkClient-EventThread-16-fed2043:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [clickview]
          15:53:15.684 [ZkClient-EventThread-16-fed2043:2181] ERROR org.apache.s4.core.Server - Could not load s4 application form s4r file [{/tmp/tmp1117261642880164701s4r}]
          com.google.inject.ProvisionException: Guice provision errors:

          1) Error injecting constructor, org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:57)
          while locating org.apache.s4.comm.topology.ClustersFromZK
          while locating org.apache.s4.comm.topology.Clusters
          for parameter 2 at org.apache.s4.core.RemoteSenders.<init>(RemoteSenders.java:66)
          while locating org.apache.s4.core.RemoteSenders
          for field at org.apache.s4.core.App.remoteSenders(App.java:49)
          while locating matrix.MatrixApp

          1 error
          at com.google.inject.internal.Errors.throwProvisionExceptionIfErrorsExist(Errors.java:451) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:65) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:944) ~[guice-3.0.jar:na]
          at org.apache.s4.core.Server.loadApp(Server.java:123) ~[s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:128) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168) [s4-core-0.5.0-incubating.jar:0.5.0-incubating]
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547) [zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) [zkclient-0.1.jar:na]
          Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:876) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:99) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:93) ~[zkclient-0.1.jar:na]
          at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:81) ~[zkclient-0.1.jar:na]
          at org.apache.s4.comm.topology.ZkClient.<init>(ZkClient.java:49) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK.<init>(ClustersFromZK.java:61) ~[s4-comm-0.5.0-incubating.jar:0.5.0-incubating]
          at org.apache.s4.comm.topology.ClustersFromZK$$FastClassByGuice$$7a883668.newInstance(<generated>) ~[guice-3.0.jar:0.5.0-incubating]
          at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na]
          at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) ~[guice-3.0.jar:na]
          at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na]
          at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:76) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:73) ~[guice-3.0.jar:na]
          at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:60) ~[guice-3.0.jar:na]
          at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:945) ~[guice-3.0.jar:na]
          ... 5 common frames omitted
          1 [ZkClient-EventThread-16-fed2043:2181] ERROR org.I0Itec.zkclient.ZkEventThread - Error handling event ZkEvent[Data of /s4/clusters/cluster11/app/s4App changed sent to org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener@16f3a299]
          org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [myApp] from URI [file:/tmp/myApp/build/libs/myApp.s4r] : cannot start application
          at org.apache.s4.deploy.DistributedDeploymentManager.deployApplication(DistributedDeploymentManager.java:134)
          at org.apache.s4.deploy.DistributedDeploymentManager$AppChangeListener.handleDataChange(DistributedDeploymentManager.java:168)
          at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:547)
          at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
          Aimee Cheng created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Aimee Cheng
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development