Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-18588

Kafka consumer on any exception should repoll records after the committed offset

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.18.0
    • 3.18.3, 3.20.0
    • camel-kafka
    • None
    • Novice

    Description

      In Camel-Kafka , if max poll records is set to more than 1 and breakOnFirstError is set to TRUE, then on any exception thrown on the first message in the batch will cause the KafkaConsumer to start repolling from zero offset because partitionLastOffset is -1 for the first message

        private boolean processException(
                  Exchange exchange, TopicPartition partition, long partitionLastOffset,
                  ExceptionHandler exceptionHandler) {
      
              // processing failed due to an unhandled exception, what should we do
              if (configuration.isBreakOnFirstError()) {
                  // we are failing and we should break out
                  if (LOG.isWarnEnabled()) {
                      LOG.warn("Error during processing {} from topic: {}", exchange, partition.topic(), exchange.getException());
                      LOG.warn("Will seek consumer to offset {} and start polling again.", partitionLastOffset);
                  }
      
                  // force commit, so we resume on next poll where we failed
                  commitManager.forceCommit(partition, partitionLastOffset);
      
                  // continue to next partition
                  return true;
              } else {
                  // will handle/log the exception and then continue to next
                  exceptionHandler.handleException("Error during processing", exchange, exchange.getException());
              }
      
              return false;
          }
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            orpiske Otavio Rodolfo Piske
            geek.rupam rupam
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment