[KAFKA-3730] Problem when updating from 0.8.2 to 0.9.0

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 0.8.2.1, 0.9.0.0
• Labels: newbie
• Environment: SUSE SLE 10.3 64bit
• Assignee: Unassigned
• Reporter: Clint Hillerman
• Created: 18/May/16 22:52
• Updated: 20/May/16 16:56
• Resolved: 20/May/16 16:56

Description

Hello,

I'm having trouble upgrading a 3 node kafka cluster from 0.8.2.1 to 0.9.0.0. I have followed the steps in the upgrade guide here:

http://kafka.apache.org/documentation.html

Also, my zookeepers are on the same box as kafka. Each node is both a zookeeper and a broker.

Here's what I did:

On each box one at a time I,

• stopped kafka.
• replaced the code with the new version. Just removed the old kafka dir and untared the new 0.9.0.0 version into it's place. Note: the data dir is in a different location and was not deleted.
• copied the server.properties file from the 0.8.2.1 version to the 0.9.0.0 config dir.
• added the "inter.broker.protocol.version=0.8.2.X" line to the server.properties in 0.9.0.0's config dir.
• restarted kafka

After I completed that process on all 3 broker/zookeeper boxes, I switched the version to 0.9.0.0 in the server.properties on one broker and restarted kafka.

This caused an error in my server.log. About one every few seconds:

[2016-05-18 15:00:27,956] WARN [ReplicaFetcherThread-0-12], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@45597bba. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading field 'topic': java.nio.BufferUnderflowException (kafka.server.ReplicaFetcherThread)

A few other things I tried:

Restarting zookeepers. There status was also correct when I ran "server mapr-zookeeper" qstatus.

The same process with 9.1 and got this error instead:
[2016-05-18 14:07:15,545] WARN [ReplicaFetcherThread-0-12], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@484ad173. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1078124, only 176 bytes available (kafka.server.ReplicaFetcherThread)

Restarting everything at once (all broker and zookeeper processes)

Please let me know if I should provide more information or if posted this in the wrong location. I'm also not sure if this is the right location to post bugs like this. If there is a forum or something where this is more appropriate please point in that direction.

Thanks,
cmhillerman