[KAFKA-8724] log cleaner thread dies when attempting to clean a __consumer_offsets partition after upgrade from 2.0->2.3

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1
• Component/s: log cleaner
• Environment: Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Assignee: Jason Gustafson
• Reporter: Keith So
• Created: 29/Jul/19 02:59
• Updated: 05/Sep/19 16:38
• Resolved: 05/Sep/19 16:38

Description

We are attempting an upgrade from Kafka 2.0 to 2.3 on a single cluster setup.  We have a mixture of Java/C++ and Python clients (Python clients are using kafka-python libraries).

After the upgrade, the log cleaner occasionally dies with the attached stack trace.  Using timestamp correlation, we pinned it down to the cleaning of a __consumer_offsets partition.  The config logged at initialization shows that 

inter.broker.protocol.version = 2.3-IV1

log.message.format.version = 2.3-IV1

We initially thought this was to do with unclean upgrade from 2.0 to 2.3, but after resetting the consumer offsets topic (via https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2)  this still recurs on initially empty consumer offset partitions.

At the moment we are working around by toggling log.cleaner.threads option using dynamic broker configuration to restore the log cleaner threads

Issue Links

• links to: GitHub Pull Request #7264