Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2343

New Kafka spout can stop emitting tuples if more than maxUncommittedOffsets tuples fail at once

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 1.1.0
    • Fix Version/s: 2.0.0, 1.1.1, 1.2.0
    • Component/s: storm-kafka-client
    • Labels:
      None

      Description

      It doesn't look like the spout is respecting maxUncommittedOffsets in all cases. If the underlying consumer returns more records in a call to poll() than maxUncommittedOffsets, they will all be added to waitingToEmit. Since poll may return up to 500 records by default (Kafka 0.10.1.1), this is pretty likely to happen with low maxUncommittedOffsets.

      The spout only checks for tuples to retry if it decides to poll, and it only decides to poll if numUncommittedOffsets < maxUncommittedOffsets. Since maxUncommittedOffsets isn't being respected when retrieving or emitting records, numUncommittedOffsets can be much larger than maxUncommittedOffsets. If more than maxUncommittedOffsets messages fail, this can cause the spout to stop polling entirely.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Srdo Stig Rohde Døssing
                Reporter:
                Srdo Stig Rohde Døssing
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 14h 50m
                  14h 50m