Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.0.0
-
None
-
None
Description
We have a 14 node kafka(2.0.0) cluster
In our case
Leader : Broker Id : 1003 Ip: 10.84.198.238
Replica : Broker Id : 1014 Ip: 10.22.2.74
A request was sent from replica -> leader to which leader(10.84.198.238) had the following exception
var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:13:04,386] ERROR Closing socket for 10.84.198.238:6667-10.22.2.74:53118-121025 because of error (kafka.network.Processor)
/var/log/kafka/server.log.2019-12-26-00-org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: FETCH, apiVersion: 8, connectionId: 10.84.198.238:6667-10.22.2.74:53118-121025, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
/var/log/kafka/server.log.2019-12-26-00-Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'forgotten_topics_data': Error reading array of size 23668, only 69 bytes available
/var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
/var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:290)
/var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:63)
------------------------------------------------------------------------
In response to this, replica (10.22.2.74) had the following log in it
[2019-12-26 00:13:04,390] WARN [ReplicaFetcher replicaId=1014, leaderId=1003, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=1014, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={topic_name=(offset=50344687, logStartOffset=24957467, maxBytes=1048576)}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1747349875, epoch=183382033)) (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1003 was disconnected before the response was read
at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:96)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:240)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:43)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:149)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:114)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
------------------------------------------------------------------------
Post this broker 1003 had the following exception
/var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:16:37,828] ERROR [ReplicaFetcher replicaId=1003, leaderId=1014, fetcherId=0] Found invalid messages during etch for partition topic_name offset 91200983 (kafka.server.ReplicaFetcherThread)
/var/log/kafka/server.log.2019-12-26-00-org.apache.kafka.common.record.InvalidRecordException: Record is corrupt (stored crc = 1460037823, computed crc = 114378201)
/var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:16:40,690] ERROR Closing socket for 10.84.198.238:6667-10.22.2.74:49850-740543 because of error (kafka.network.Processor)
/var/log/kafka/server.log.2019-12-26-00-org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: FETCH, apiVersion: 8, connectionId: 10.84.198.238:6667-10.22.2.74:49850-740543, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
Could you help us with the above issue?