Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2546

Kafka spout can stall / get stuck due to edge case with failing tuples

    Details

      Description

      The mechanism for replaying a failed tuple involves seeking the kafka consumer to the failing offset and then re-emitting it into the topology. A tuple, when emitted the first time, will have an entry created in OffsetManager. This entry will be removed only after the tuple is successfully acknowledged and its offset successfully committed. Till then, commits for offsets beyond the failing offset for that TopicPartition will be blocked.

      It is possible that when the spout seeks the consumer to the failing offset, the corresponding kafka message is not returned in the poll response. This can happen due to that offset being deleted or compacted away. In this scenario that partition will be blocked from committing and progressing.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Srdo Stig Rohde Døssing
                Reporter:
                ranganp Prasanna Ranganathan
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m