[STORM-3128] Connection refused error in AsyncLocalizerTest

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: storm-server
• Assignee: Unassigned
• Reporter: Zhengdai Hu
• Created: 27/Jun/18 18:12
• Updated: 08/Aug/18 16:59

Description

In AsyncLocalizerTest testKeyNotFoundException, a localBlobStore is created and tries but failed to connect to zookeeper due to connection error. I'm not sure if this compromises the test even though it is passed after connection retry timeout. But it's nice to keep in mind.

2018-06-27 13:05:28.005 [main-SendThread(localhost:2181)] INFO  org.apache.storm.shade.org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-27 13:05:28.032 [main] INFO  org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2018-06-27 13:05:28.035 [main-SendThread(localhost:2181)] WARN  org.apache.storm.shade.org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_171]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_171]
	at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]

I managed to track down the source where the exception is thrown, but it's really strange that this is called by a StormTimer inside Supervisor, which is not declared anywhere in this test. I'm completely lost by now.

2018-08-08 11:45:30.217 [heartbeatTimer] ERROR org.apache.storm.zookeeper.ClientZookeeper - e: {}
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /supervisors
        at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:268) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:257) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:254) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:247) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:206) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35) ~[shaded-deps-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.zookeeper.ClientZookeeper.existsNode(ClientZookeeper.java:145) [storm-client-2.0.0-SNAPSHOT.jar:?]
        at org.apache.storm.zookeeper.ClientZookeeper.mkdirsImpl(ClientZookeeper.java:292) [storm-client-2.0.0-SNAPSHOT.jar:?]
        at org.apache.storm.zookeeper.ClientZookeeper.mkdirs(ClientZookeeper.java:70) [storm-client-2.0.0-SNAPSHOT.jar:?]
        at org.apache.storm.cluster.ZKStateStorage.set_ephemeral_node(ZKStateStorage.java:129) [storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.cluster.StormClusterStateImpl.supervisorHeartbeat(StormClusterStateImpl.java:522) [storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.apache.storm.daemon.supervisor.timer.SupervisorHeartbeat.run(SupervisorHeartbeat.java:96) [classes/:?]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.0.0-SNAPSHOT.jar:?]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.0.0-SNAPSHOT.jar:?]

Issue Links

• blocks: STORM-3173 flush metrics to ScheduledReporter on shutdown
• is related to: STORM-3125 Refactoring methods in components for Supervisor and DRPC