XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: storm-core
    • Labels:
      None

      Description

      test-load at netty-unit-test occasionally fails.

      Running backtype.storm.messaging.netty-unit-test
      Tests run: 5, Passed: 20, Failures: 0, Errors: 1
      
      --------------------------------------------------
      classname: backtype.storm.messaging.netty-unit-test / testname: test-load
      Uncaught exception, not in assertion.
      expected: nil
        actual: java.lang.AssertionError: Test timed out (5000ms) (empty? (.getLoad client [(int 1) (int 2)]))
       at backtype.storm.messaging.netty_unit_test$test_load_fn.invoke (netty_unit_test.clj:115)
          backtype.storm.messaging.netty_unit_test/fn (netty_unit_test.clj:144)
          clojure.test$test_var$fn__7670.invoke (test.clj:704)
          clojure.test$test_var.invoke (test.clj:704)
          clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
          clojure.test$default_fixture.invoke (test.clj:674)
          clojure.test$test_vars$fn__7692.invoke (test.clj:722)
          clojure.test$default_fixture.invoke (test.clj:674)
          clojure.test$test_vars.invoke (test.clj:718)
          clojure.test$test_all_vars.invoke (test.clj:728)
          clojure.test$test_ns.invoke (test.clj:747)
          clojure.core$map$fn__4553.invoke (core.clj:2624)
          clojure.lang.LazySeq.sval (LazySeq.java:40)
          clojure.lang.LazySeq.seq (LazySeq.java:49)
          clojure.lang.Cons.next (Cons.java:39)
          clojure.lang.RT.boundedLength (RT.java:1735)
          clojure.lang.RestFn.applyTo (RestFn.java:130)
          clojure.core$apply.invoke (core.clj:632)
          clojure.test$run_tests.doInvoke (test.clj:762)
          clojure.lang.RestFn.invoke (RestFn.java:408)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900$fn__10901.invoke (test_runner.clj:107)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900.invoke (test_runner.clj:53)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899.invoke (test_runner.clj:52)
          clojure.lang.LazySeq.sval (LazySeq.java:40)
          clojure.lang.LazySeq.seq (LazySeq.java:49)
          clojure.lang.RT.seq (RT.java:507)
          clojure.core/seq (core.clj:137)
          clojure.core$dorun.invoke (core.clj:3009)
          org.apache.storm.testrunner$eval10893.invoke (test_runner.clj:52)
          clojure.lang.Compiler.eval (Compiler.java:6782)
          clojure.lang.Compiler.load (Compiler.java:7227)
          clojure.lang.Compiler.loadFile (Compiler.java:7165)
          clojure.main$load_script.invoke (main.clj:275)
          clojure.main$script_opt.invoke (main.clj:337)
          clojure.main$main.doInvoke (main.clj:421)
          clojure.lang.RestFn.invoke (RestFn.java:421)
          clojure.lang.Var.invoke (Var.java:383)
          clojure.lang.AFn.applyToHelper (AFn.java:156)
          clojure.lang.Var.applyTo (Var.java:700)
          clojure.main.main (main.java:37)
            at: test_runner.clj:105
      -------------------- system-out --------------------
      30464 [main] INFO  b.s.m.netty-unit-test - 2 test load
      30464 [main] INFO  b.s.m.TransportFactory - Storm peer transport plugin:backtype.storm.messaging.netty.Context
      30474 [main] INFO  b.s.m.n.Server - Create Netty Server Netty-server-localhost-6700, buffer_size: 1024, maxWorkers: 1
      30476 [main] INFO  b.s.m.n.Client - creating Netty Client, connecting to localhost:6700, bufferSize: 1024
      30476 [main] INFO  b.s.m.netty-unit-test - Waiting until all Netty connections are ready...
      30588 [main] INFO  b.s.m.netty-unit-test - All Netty connections are ready
      30603 [main] INFO  b.s.m.n.Client - closing Netty Client Netty-Client-localhost/127.0.0.1:6700
      30603 [main] INFO  b.s.m.n.Client - waiting up to 600000 ms to send 0 pending messages to Netty-Client-localhost/127.0.0.1:6700
      30605 [main] INFO  b.s.m.netty-unit-test - 2 test load
      30605 [main] INFO  b.s.m.TransportFactory - Storm peer transport plugin:backtype.storm.messaging.netty.Context
      30609 [main] INFO  b.s.m.n.Server - Create Netty Server Netty-server-localhost-6700, buffer_size: 1024, maxWorkers: 1
      30610 [main] INFO  b.s.m.n.Client - creating Netty Client, connecting to localhost:6700, bufferSize: 1024
      30610 [main] INFO  b.s.m.netty-unit-test - Waiting until all Netty connections are ready...
      30722 [main] INFO  b.s.m.netty-unit-test - All Netty connections are ready
      30799 [Thread-147-SendThread(localhost:2000)] WARN  o.a.z.ClientCnxn - Session 0x151ec0a79640011 for server null, unexpected error, closing socket connection and attempting reconnect
      java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_31]
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_31]
      	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
      	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [zookeeper-3.4.6.jar:3.4.6-1569965]
      32859 [Thread-147-SendThread(localhost:2000)] WARN  o.a.z.ClientCnxn - Session 0x151ec0a79640011 for server null, unexpected error, closing socket connection and attempting reconnect
      java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_31]
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_31]
      	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
      	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [zookeeper-3.4.6.jar:3.4.6-1569965]
      34645 [Thread-147-SendThread(localhost:2000)] WARN  o.a.z.ClientCnxn - Session 0x151ec0a79640011 for server null, unexpected error, closing socket connection and attempting reconnect
      java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_31]
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_31]
      	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
      	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [zookeeper-3.4.6.jar:3.4.6-1569965]
      35727 [main] INFO  b.s.m.netty-unit-test - Condition (empty? (.getLoad client [(int 1) (int 2)])) not met in 5000ms
      35727 [main] INFO  b.s.m.netty-unit-test - "pool-305-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
              at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at java.lang.Thread.run(Thread.java:745)
      "client-schedule-service-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
              at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at java.lang.Thread.run(Thread.java:745)
      "Netty-server-localhost-6700-boss-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "Netty-server-localhost-6700-worker-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "client-boss-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "client-worker-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-147-EventThread" 
         java.lang.Thread.State: WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
      "Thread-147-SendThread(localhost:2000)" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
              at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
              at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
      "pool-299-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Curator-Framework-0" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:795)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:63)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "Curator-ConnectionStateManager-0" 
         java.lang.Thread.State: WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
              at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
              at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
              at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-254-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "pool-211-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-123" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-121" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-120" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "pool-168-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-167-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-77" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-75" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-74" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-72" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-7" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-6" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-5" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-4" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-3" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-2" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-117-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-116-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "metric/stat timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              at java.util.TimerThread.run(Timer.java:505)
      "disruptor-flush-trigger" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.util.TimerThread.mainLoop(Timer.java:526)
              at java.util.TimerThread.run(Timer.java:505)
      "Thread-29" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-24" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-19" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-17" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-16" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-14" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "clojure-agent-send-off-pool-0" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "FileBlobStore cleanup thread" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              at java.util.TimerThread.run(Timer.java:505)
      "Thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Signal Dispatcher" 
         java.lang.Thread.State: RUNNABLE
      "Finalizer" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      "Reference Handler" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
      "main" 
         java.lang.Thread.State: RUNNABLE
              at sun.management.ThreadImpl.getThreadInfo1(Native Method)
              at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
              at backtype.storm.utils.Utils.threadDump(Utils.java:1134)
              at backtype.storm.messaging.netty_unit_test$test_load_fn.invoke(netty_unit_test.clj:115)
              at backtype.storm.messaging.netty_unit_test$fn__938.invoke(netty_unit_test.clj:144)
              at clojure.test$test_var$fn__7670.invoke(test.clj:704)
              at clojure.test$test_var.invoke(test.clj:704)
              at clojure.test$test_vars$fn__7692$fn__7697.invoke(test.clj:722)
              at clojure.test$default_fixture.invoke(test.clj:674)
              at clojure.test$test_vars$fn__7692.invoke(test.clj:722)
              at clojure.test$default_fixture.invoke(test.clj:674)
              at clojure.test$test_vars.invoke(test.clj:718)
              at clojure.test$test_all_vars.invoke(test.clj:728)
              at clojure.test$test_ns.invoke(test.clj:747)
              at clojure.core$map$fn__4553.invoke(core.clj:2624)
              at clojure.lang.LazySeq.sval(LazySeq.java:40)
              at clojure.lang.LazySeq.seq(LazySeq.java:49)
              at clojure.lang.Cons.next(Cons.java:39)
              at clojure.lang.RT.boundedLength(RT.java:1735)
              at clojure.lang.RestFn.applyTo(RestFn.java:130)
              at clojure.core$apply.invoke(core.clj:632)
              at clojure.test$run_tests.doInvoke(test.clj:762)
              at clojure.lang.RestFn.invoke(RestFn.java:408)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900$fn__10901.invoke(test_runner.clj:107)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900.invoke(test_runner.clj:53)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899.invoke(test_runner.clj:52)
              at clojure.lang.LazySeq.sval(LazySeq.java:40)
              at clojure.lang.LazySeq.seq(LazySeq.java:49)
              at clojure.lang.RT.seq(RT.java:507)
              at clojure.core$seq__4128.invoke(core.clj:137)
              at clojure.core$dorun.invoke(core.clj:3009)
              at org.apache.storm.testrunner$eval10893.invoke(test_runner.clj:52)
              at clojure.lang.Compiler.eval(Compiler.java:6782)
              at clojure.lang.Compiler.load(Compiler.java:7227)
              at clojure.lang.Compiler.loadFile(Compiler.java:7165)
              at clojure.main$load_script.invoke(main.clj:275)
              at clojure.main$script_opt.invoke(main.clj:337)
              at clojure.main$main.doInvoke(main.clj:421)
              at clojure.lang.RestFn.invoke(RestFn.java:421)
              at clojure.lang.Var.invoke(Var.java:383)
              at clojure.lang.AFn.applyToHelper(AFn.java:156)
              at clojure.lang.Var.applyTo(Var.java:700)
              at clojure.main.main(main.java:37)
      

      and another stack trace (a bit different)

      classname: backtype.storm.messaging.netty-unit-test / testname: test-load
      Uncaught exception, not in assertion.
      expected: nil
        actual: java.lang.AssertionError: Test timed out (5000ms) (empty? (.getLoad client [(int 1) (int 2)]))
       at backtype.storm.messaging.netty_unit_test$test_load_fn.invoke (netty_unit_test.clj:115)
          backtype.storm.messaging.netty_unit_test/fn (netty_unit_test.clj:144)
          clojure.test$test_var$fn__7670.invoke (test.clj:704)
          clojure.test$test_var.invoke (test.clj:704)
          clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
          clojure.test$default_fixture.invoke (test.clj:674)
          clojure.test$test_vars$fn__7692.invoke (test.clj:722)
          clojure.test$default_fixture.invoke (test.clj:674)
          clojure.test$test_vars.invoke (test.clj:718)
          clojure.test$test_all_vars.invoke (test.clj:728)
          clojure.test$test_ns.invoke (test.clj:747)
          clojure.core$map$fn__4553.invoke (core.clj:2624)
          clojure.lang.LazySeq.sval (LazySeq.java:40)
          clojure.lang.LazySeq.seq (LazySeq.java:49)
          clojure.lang.Cons.next (Cons.java:39)
          clojure.lang.RT.boundedLength (RT.java:1735)
          clojure.lang.RestFn.applyTo (RestFn.java:130)
          clojure.core$apply.invoke (core.clj:632)
          clojure.test$run_tests.doInvoke (test.clj:762)
          clojure.lang.RestFn.invoke (RestFn.java:408)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900$fn__10901.invoke (test_runner.clj:107)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900.invoke (test_runner.clj:53)
          org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899.invoke (test_runner.clj:52)
          clojure.lang.LazySeq.sval (LazySeq.java:40)
          clojure.lang.LazySeq.seq (LazySeq.java:49)
          clojure.lang.RT.seq (RT.java:507)
          clojure.core/seq (core.clj:137)
          clojure.core$dorun.invoke (core.clj:3009)
          org.apache.storm.testrunner$eval10893.invoke (test_runner.clj:52)
          clojure.lang.Compiler.eval (Compiler.java:6782)
          clojure.lang.Compiler.load (Compiler.java:7227)
          clojure.lang.Compiler.loadFile (Compiler.java:7165)
          clojure.main$load_script.invoke (main.clj:275)
          clojure.main$script_opt.invoke (main.clj:337)
          clojure.main$main.doInvoke (main.clj:421)
          clojure.lang.RestFn.invoke (RestFn.java:421)
          clojure.lang.Var.invoke (Var.java:383)
          clojure.lang.AFn.applyToHelper (AFn.java:156)
          clojure.lang.Var.applyTo (Var.java:700)
          clojure.main.main (main.java:37)
            at: test_runner.clj:105
      -------------------- system-out --------------------
      35043 [main] INFO  b.s.m.netty-unit-test - 2 test load
      35044 [main] INFO  b.s.m.TransportFactory - Storm peer transport plugin:backtype.storm.messaging.netty.Context
      35049 [main] INFO  b.s.m.n.Server - Create Netty Server Netty-server-localhost-6700, buffer_size: 1024, maxWorkers: 1
      35050 [main] INFO  b.s.m.n.Client - creating Netty Client, connecting to localhost:6700, bufferSize: 1024
      35051 [main] INFO  b.s.m.netty-unit-test - Waiting until all Netty connections are ready...
      35173 [main] INFO  b.s.m.netty-unit-test - All Netty connections are ready
      35188 [main] INFO  b.s.m.n.Client - closing Netty Client Netty-Client-localhost/127.0.0.1:6700
      35188 [main] INFO  b.s.m.n.Client - waiting up to 600000 ms to send 0 pending messages to Netty-Client-localhost/127.0.0.1:6700
      35192 [main] INFO  b.s.m.netty-unit-test - 2 test load
      35192 [main] INFO  b.s.m.TransportFactory - Storm peer transport plugin:backtype.storm.messaging.netty.Context
      35213 [main] INFO  b.s.m.n.Server - Create Netty Server Netty-server-localhost-6700, buffer_size: 1024, maxWorkers: 1
      35213 [main] INFO  b.s.m.n.Client - creating Netty Client, connecting to localhost:6700, bufferSize: 1024
      35214 [main] INFO  b.s.m.netty-unit-test - Waiting until all Netty connections are ready...
      35330 [main] INFO  b.s.m.netty-unit-test - All Netty connections are ready
      35344 [client-worker-1] INFO  b.s.m.n.StormClientHandler - Connection to localhost/127.0.0.1:6700 failed:
      java.lang.ClassCastException: java.util.ArrayList cannot be cast to backtype.storm.messaging.netty.SaslMessageToken
      	at backtype.storm.messaging.netty.SaslStormClientHandler.messageReceived(SaslStormClientHandler.java:115) ~[classes/:?]
      	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.9.0.Final.jar:?]
      	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.9.0.Final.jar:?]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_31]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_31]
      	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
      40352 [main] INFO  b.s.m.netty-unit-test - Condition (empty? (.getLoad client [(int 1) (int 2)])) not met in 5000ms
      40352 [main] INFO  b.s.m.netty-unit-test - "pool-303-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
              at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at java.lang.Thread.run(Thread.java:745)
      "client-schedule-service-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
              at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at java.lang.Thread.run(Thread.java:745)
      "Netty-server-localhost-6700-boss-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "Netty-server-localhost-6700-worker-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "client-boss-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "client-worker-1" 
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-297-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-169" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "pool-254-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-143" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "pool-211-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "pool-168-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-167-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-75" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "pool-117-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-8" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-7" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-6" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-5" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-4" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-3" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-2" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "pool-116-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "reset-log-levels-timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.timer$mk_timer$fn__4208$fn__4209.invoke(timer.clj:60)
              at backtype.storm.timer$mk_timer$fn__4208.invoke(timer.clj:42)
              at clojure.lang.AFn.run(AFn.java:22)
              at java.lang.Thread.run(Thread.java:745)
      "pool-113-thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "metric/stat timer" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              at java.util.TimerThread.run(Timer.java:505)
      "disruptor-flush-trigger" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.util.TimerThread.mainLoop(Timer.java:526)
              at java.util.TimerThread.run(Timer.java:505)
      "Thread-19" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-17" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-16" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Thread-14" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "clojure-agent-send-off-pool-0" 
         java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "FileBlobStore cleanup thread" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              at java.util.TimerThread.run(Timer.java:505)
      "Thread-1" 
         java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at backtype.storm.utils.Time.sleepUntil(Time.java:86)
              at backtype.storm.utils.Time.sleep(Time.java:91)
              at backtype.storm.utils.TimeCacheMap$1.run(TimeCacheMap.java:61)
              at java.lang.Thread.run(Thread.java:745)
      "Signal Dispatcher" 
         java.lang.Thread.State: RUNNABLE
      "Finalizer" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      "Reference Handler" 
         java.lang.Thread.State: WAITING
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
      "main" 
         java.lang.Thread.State: RUNNABLE
              at sun.management.ThreadImpl.getThreadInfo1(Native Method)
              at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
              at backtype.storm.utils.Utils.threadDump(Utils.java:1134)
              at backtype.storm.messaging.netty_unit_test$test_load_fn.invoke(netty_unit_test.clj:115)
              at backtype.storm.messaging.netty_unit_test$fn__938.invoke(netty_unit_test.clj:144)
              at clojure.test$test_var$fn__7670.invoke(test.clj:704)
              at clojure.test$test_var.invoke(test.clj:704)
              at clojure.test$test_vars$fn__7692$fn__7697.invoke(test.clj:722)
              at clojure.test$default_fixture.invoke(test.clj:674)
              at clojure.test$test_vars$fn__7692.invoke(test.clj:722)
              at clojure.test$default_fixture.invoke(test.clj:674)
              at clojure.test$test_vars.invoke(test.clj:718)
              at clojure.test$test_all_vars.invoke(test.clj:728)
              at clojure.test$test_ns.invoke(test.clj:747)
              at clojure.core$map$fn__4553.invoke(core.clj:2624)
              at clojure.lang.LazySeq.sval(LazySeq.java:40)
              at clojure.lang.LazySeq.seq(LazySeq.java:49)
              at clojure.lang.Cons.next(Cons.java:39)
              at clojure.lang.RT.boundedLength(RT.java:1735)
              at clojure.lang.RestFn.applyTo(RestFn.java:130)
              at clojure.core$apply.invoke(core.clj:632)
              at clojure.test$run_tests.doInvoke(test.clj:762)
              at clojure.lang.RestFn.invoke(RestFn.java:408)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900$fn__10901.invoke(test_runner.clj:107)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899$fn__10900.invoke(test_runner.clj:53)
              at org.apache.storm.testrunner$eval10893$iter__10894__10898$fn__10899.invoke(test_runner.clj:52)
              at clojure.lang.LazySeq.sval(LazySeq.java:40)
              at clojure.lang.LazySeq.seq(LazySeq.java:49)
              at clojure.lang.RT.seq(RT.java:507)
              at clojure.core$seq__4128.invoke(core.clj:137)
              at clojure.core$dorun.invoke(core.clj:3009)
              at org.apache.storm.testrunner$eval10893.invoke(test_runner.clj:52)
              at clojure.lang.Compiler.eval(Compiler.java:6782)
              at clojure.lang.Compiler.load(Compiler.java:7227)
              at clojure.lang.Compiler.loadFile(Compiler.java:7165)
              at clojure.main$load_script.invoke(main.clj:275)
              at clojure.main$script_opt.invoke(main.clj:337)
              at clojure.main$main.doInvoke(main.clj:421)
              at clojure.lang.RestFn.invoke(RestFn.java:421)
              at clojure.lang.Var.invoke(Var.java:383)
              at clojure.lang.AFn.applyToHelper(AFn.java:156)
              at clojure.lang.Var.applyTo(Var.java:700)
              at clojure.main.main(main.java:37)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kabhwan Jungtaek Lim
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: