[STORM-292] emit blocks the publishing bolt if disrupter queue is full

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2-incubating
• Fix Version/s: 0.10.0
• Component/s: storm-core
• Assignee: Robert Joseph Evans
• Reporter: Kishor Patil
• Created: 22/Apr/14 17:43
• Updated: 06/Apr/17 13:28
• Resolved: 08/Sep/15 20:47

Description

During testing, i notice once the disruptor queue is full, it blocks (timed_wait ) the publishing bolt essentially creating slowdown live-lock. Should the outputCollector.emit be a non-blocking call?

Also, configs for better control on the disruptor queue/buffer size seem to not have any impact.

"topology.executor.receive.buffer.size"
"topology.receiver.buffer.size"
"topology.executor.send.buffer.size"
"topology.transfer.buffer.size"

Below is example of topology that re-creates live lock scenario with disruptor queue.

https://github.com/kishorvpatil/incubator-storm/blob/dqueue-full/examples/storm-starter/src/jvm/storm/starter/SplitJoinTopology.java

"Thread-9-splitjoinbolt" prio=10 tid=0x00007fe518a9f000 nid=0x64b5 waiting on condition [0x00007fe5144f2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
at com.lmax.disruptor.SingleThreadedClaimStrategy.waitForFreeSlotAt(SingleThreadedClaimStrategy.java:129)
at com.lmax.disruptor.SingleThreadedClaimStrategy.incrementAndGet(SingleThreadedClaimStrategy.java:81)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:113)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:51)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4913.invoke(executor.clj:176)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4913.invoke(executor.clj:183)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4913.invoke(executor.clj:185)
at backtype.storm.daemon.executor$fn_5141$fn5155$bolt_emit_5184.invoke(executor.clj:683)
at backtype.storm.daemon.executor$fn_5141$fn$reify_5190.emit(executor.clj:693)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:186)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:32)
at backtype.storm.topology.BasicOutputCollector.emit(BasicOutputCollector.java:19)
at storm.starter.bolt.SplitAndCountBolt.execute(SplitAndCountBolt.java:24)
at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:33)
at backtype.storm.daemon.executor$fn_5141$tuple_action_fn_5143.invoke(executor.clj:634)

Issue Links

• Is contained by: STORM-216 Secure Multi-tenant Storm