Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-13338

ConsumerRebalanceListener is not registered when topicIsPattern is turned off. Causing message loss or too many duplicates

    XMLWordPrintableJSON

    Details

    • Estimated Complexity:
      Unknown

      Description

      Not registering ConsumerRebalanceListener when topicIsPattern turned off, causes either too many duplicate messages getting delivered to message processing layer or causes message loss depending on autoOffsetReset flag value.
       
      Issue is noticed with below test scenario
      Topic: e.g. TestMessage
      Number of partitions: 25 (this is not completely relevant to issue)
      ConsumerUri: kafka:<kafka-broker-uris>?topic=TestMessage&groupId=TestMessage_GROUP&consumersCount=5&autoCommitEnable=false&offsetRepository=#<DB-state-repository>
      Number processes running consumers: 2 or more; e.g. two instances of Docker containers listening from same topic each with 5 consumer threads
       
      Please note that offset management is done in DB with an implementation of Camel's StateRepository interface to eliminate too duplicate messages arriving at the message processor (at most one message processing needed for my use case)
       
      When a second process instance (say second container) brought up, since ConsumerRebalanceListener is not registered, partitions start processing at earliest offsets if autoOffsetReset is set to earliest as expected. In this case, too many duplicate messages arrive at the message processing layer. If autoOffsetReset is set to latest, message loss occurs for certain partitions. If autoOffsetReset is set to none, no valid offset exception thrown by broker
       
      Issue would be noticed consistently when simulating slow message processing for the above test scenario
       
       

        Attachments

          Activity

            People

            • Assignee:
              davsclaus Claus Ibsen
              Reporter:
              viswaram Viswa Ramamoorthy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m