Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2994

KafkaSpout consumes messages but doesn't commit offsets

    Details

      Description

      A topology that consumes from two different Kafka clusters: 0.10.1.1 and 0.10.2.1.

      Spouts consuming from 0.10.2.1 have a low lag (and regularly commit offsets) 

      The Spout that consumes from 0.10.1.1 exhibits either:

      1- Unknown lag

      2- Lag that increments as the Spout reads messages from Kafka

       

      In DEBUG, Offset manager logs: "topic-partition has NO offsets ready to be committed", despite continuing to consume messages.

      Several configuration tweaks were tried, including setting maxRetries to 1, in case messages with a lower offset were being retried (logs didn't show it, though)

      offsetCommitPeriodMs was also  lowered to no avail.

      The only configuration that works is to have ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG=true, but this is undesired   since we lose processing guarantees.

       

        Attachments

          Activity

            People

            • Assignee:
              RAbreu Rui Abreu
              Reporter:
              RAbreu Rui Abreu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 50m
                3h 50m