Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2087

Storm-kafka-client: Failed tuples are not always replayed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 1.0.2
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: storm-kafka-client
    • Labels:
      None

      Description

      I am working with kafka 10 and the storm-kafka-client from master. It appears that tuples are not always being replayed when they are failed.

      With a topology that randomly fails tuples a small percentage of the time I found that the committed kafka offset would get stuck and eventually processing would stop even though the committed offset was no where near the end of the topic.

      I have also replicated the issue in unit tests with this PR:
      https://github.com/apache/storm/pull/1679

      It seems that increasing the number of times I call nextTuple for the in order case will make it work, but it doesn't seem to help the case where tuples are failed out of order from which they were emitted.

        Issue Links

          Activity

          Hide
          jfenc91 Jeff Fenchel added a comment -

          The fix is now in that PR as well

          Show
          jfenc91 Jeff Fenchel added a comment - The fix is now in that PR as well
          Hide
          asmaier Andreas Maier added a comment -

          Is this a duplicate of STORM-2077 ?

          Show
          asmaier Andreas Maier added a comment - Is this a duplicate of STORM-2077 ?
          Hide
          Srdo Stig Rohde Døssing added a comment -

          Andreas Maier Yes, as far as I can tell.

          Show
          Srdo Stig Rohde Døssing added a comment - Andreas Maier Yes, as far as I can tell.
          Hide
          kabhwan Jungtaek Lim added a comment -

          Added to 1.1.0 epic since it's a major bug and patch is available (PR against master is merged.)

          Show
          kabhwan Jungtaek Lim added a comment - Added to 1.1.0 epic since it's a major bug and patch is available (PR against master is merged.)
          Hide
          kabhwan Jungtaek Lim added a comment -

          Merged into master and 1.x branches.

          Show
          kabhwan Jungtaek Lim added a comment - Merged into master and 1.x branches.

            People

            • Assignee:
              jfenc91 Jeff Fenchel
              Reporter:
              jfenc91 Jeff Fenchel
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10.5h
                10.5h

                  Development