Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2104

New Kafka spout crashes if partitions are reassigned while tuples are in-flight

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 1.1.0
    • Fix Version/s: 2.0.0, 1.1.0, 1.0.4
    • Component/s: storm-kafka
    • Labels:
      None

      Description

      The new KafkaSpout may throw NPEs if partitions are reassigned while tuples are in-flight. The ack function assumes that the spout instance is always responsible for tuples it emitted, which isn't true if partitions were reassigned since the tuple was emitted. The fail function also assumes that failed tuples should be replayed, which is useless if the tuple is for a partition the spout isn't assigned, since it then can't commit the tuple if it succeeds. Both functions should check that the spout instance is responsible for the incoming tuple before scheduling it for retry or adding it to the acked list.

        Issue Links

          Activity

          Hide
          revans2 Robert Joseph Evans added a comment -

          I merged the pull request into master branch, but it has some minor merge conflicts and java 7 incompatibilities. If you want this to go into 1.1 you will need to create a new pull request for that. Overall it is really good.

          Show
          revans2 Robert Joseph Evans added a comment - I merged the pull request into master branch, but it has some minor merge conflicts and java 7 incompatibilities. If you want this to go into 1.1 you will need to create a new pull request for that. Overall it is really good.
          Hide
          Srdo Stig Rohde Døssing added a comment -

          Thanks for the review Robert Joseph Evans. I put up a 1.x version too

          Show
          Srdo Stig Rohde Døssing added a comment - Thanks for the review Robert Joseph Evans . I put up a 1.x version too
          Hide
          kabhwan Jungtaek Lim added a comment -

          Added this to 1.1.0 epic.

          Show
          kabhwan Jungtaek Lim added a comment - Added this to 1.1.0 epic.
          Hide
          kabhwan Jungtaek Lim added a comment -

          Thanks Stig Rohde Døssing, I also merged this into 1.x branch.

          Show
          kabhwan Jungtaek Lim added a comment - Thanks Stig Rohde Døssing , I also merged this into 1.x branch.

            People

            • Assignee:
              Srdo Stig Rohde Døssing
              Reporter:
              Srdo Stig Rohde Døssing
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 20m
                4h 20m

                  Development