Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1358

Porting JStorm multi-thread mode of spout to Storm

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The "single-thread" mode is simliar to Storm while the "multi-thread" mode separates the processing of ack/fail and nextTuple to two threads. It means we can stay in nextTuple for a long time without any side effect on ack/fail.
      Let's think about an example of kafka spout. We can initiate a consumer thread for kafka when initialization of spout. Then the comsumer starts to pull events from kafka and pulish the retreived events into a local queue. At meantime, nextTuple waits to read at this queue. If any available events, nextTuple will get notification faster and flush them to downstream. This model could probably introduce better performance compared with "single-thread" mode.

      For this mode, the max pending configuration of spout might not be useful as expectation. It depends on how long we stay in nextTuple. But backpressure is a good choice to resolve flow control problem.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              basti.lj Basti Liu
              Reporter:
              basti.lj Basti Liu

              Dates

              • Created:
                Updated:

                Issue deployment