[CAMEL-14980] camel-kafka - SerializationException - consumer keeps leaving and rejoining the group

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.9.0
• Component/s: camel-kafka
• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: joseph m'bimbi-bene
• Created: 29/Apr/20 05:23
• Updated: 22/Mar/21 08:39
• Resolved: 22/Mar/21 08:24

Description

Hello everyone,

 

I found out i few days ago that if a `SerializationException` is thrown when the consumer tries to poll messages, it will keep leaving and joining the consumer-group indefinitely and without any informative log.
The exception cannot either be handled by any camel exception handler.

After some searching in the code i found out the culprit:

// org.apache.camel.component.kafka.KafkaConsumer (ligns 406-415):
catch (KafkaException e) {
  // some kind of error in kafka, it may happen during
  // unsubscribing or during normal processing
  if (unsubscribing){             
    getExceptionHandler().handleException("Error unsubscribing " + threadId + " from kafka topic " + topicName, e);   
  }else {
    LOG.debug("KafkaException consuming {} from topic {} causedby {}. Will attempt to re-connect on next run", threadId, topicName, e.getMessage());
    reConnect = true;
  }
}

 

`SerializationException` extends from `KafkaException`, but it is definitely not a recoverable exception.

It logs with debug level, which makes it hard to track, there are SO many things logging in debug.

It it cannot be handled by any camel exception handling mechanism.

I think it would be better to either:

• change that catch so that it pinpoints the subclasses of `KafkaException` that are actually recoverable from rejoining (maybe `WakeupException` and a couple others)
• add a `catch` block for `SerializationException` and maybe `ConfigException` and `OAuthBearerConfigException` before, with a log error andallow the user to handle those exceptions
• remove that catch block entirely and let users handle any KafkaException however they see fit.

Thank you

Issue Links

• links to: GitHub Pull Request #5248