Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2666

Storm-kafka-client spout can sometimes emit messages that were already committed.

    Details

      Description

      Under a certain heavy load, for failed/timeout tuples, the retry service will ack tuple for failed max times. Kafka Client Spout will commit after reached the commit interval. However seems some 'on the way' tuples will be failed again, the retry service will cause Spout to emit again, and acked eventually to OffsetManager.

      In some cases such offsets are too many, exceeding the max-uncommit, causing org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset unable to find next commit point, and Spout for this partition will not poll any more.

      By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this information helps.

      resulting logs like below. I'm not sure if the issue has already been raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.

      2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248 248] [INFO] Successful ack for tuple message [

      {topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}

      ].
      2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager Thread-23-spout-executor[248 248] [WARN] topic-partition [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed Offset [16003]

      Edit:
      See https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893 for the current best guess at the root cause of this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Srdo Stig Rohde Døssing
                Reporter:
                GuangDu Guang Du
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m