[SPARK-22991] High read latency with spark streaming 2.2.1 and kafka 0.10.0.1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.2.1
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Kiran Shivappa Japannavar
• Created: 08/Jan/18 15:18
• Updated: 16/Jan/18 13:34
• Resolved: 16/Jan/18 13:34

Description

Spark 2.2.1 + Kafka 0.10 + Spark streaming.

Batch duration is 1s, Max rate per partition is 500, poll interval is 120 seconds, max poll records is 500 and no of partitions in Kafka is 500, enabled cache consumer.

While trying to read data from Kafka we are observing very high read latencies intermittently.The high latencies results in Kafka consumer session expiration and hence the Kafka brokers removes the consumer from the group. The consumer keeps retrying and finally fails with the

[org.apache.kafka.clients.NetworkClient] - Disconnecting from node 12 due to request timeout
[org.apache.kafka.clients.NetworkClient] - Cancelled request ClientRequest
[org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient] - Cancelled FETCH request ClientRequest.**
Due to this a lot of batches are in the queued state.

The high read latencies are occurring whenever multiple clients are parallelly trying to read the data from the same Kafka cluster. The Kafka cluster is having a large number of brokers and can support high network bandwidth.

When running with spark 1.5 and Kafka 0.8 consumer client against the same Kafka cluster we are not seeing any read latencies.