[STORM-1358] Porting JStorm multi-thread mode of spout to Storm

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: jstorm-merger
• Assignee: Basti Liu
• Reporter: Basti Liu
• Created: 01/Dec/15 02:50
• Updated: 29/Apr/16 04:15

Description

There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The "single-thread" mode is simliar to Storm while the "multi-thread" mode separates the processing of ack/fail and nextTuple to two threads. It means we can stay in nextTuple for a long time without any side effect on ack/fail.
Let's think about an example of kafka spout. We can initiate a consumer thread for kafka when initialization of spout. Then the comsumer starts to pull events from kafka and pulish the retreived events into a local queue. At meantime, nextTuple waits to read at this queue. If any available events, nextTuple will get notification faster and flush them to downstream. This model could probably introduce better performance compared with "single-thread" mode.

For this mode, the max pending configuration of spout might not be useful as expectation. It depends on how long we stay in nextTuple. But backpressure is a good choice to resolve flow control problem.

Issue Links

• is part of: STORM-1319 Port JStorm features to storm
• mentioned in: Page Loading...