[SPARK-24056] Make consumer creation lazy in Kafka source for Structured streaming

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Target Version/s: 2.4.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 23/Apr/18 21:05
• Updated: 13/Jul/18 05:27
• Resolved: 24/Apr/18 21:34

Description

Currently, the driver side of the Kafka source (i.e. KafkaMicroBatchReader) eagerly creates a consumer as soon as the Kafk aMicroBatchReader is created. However, we create dummy KafkaMicroBatchReader to get the schema and immediately stop it. Its better to make the consumer creation lazy, it will be created on the first attempt to fetch offsets using the KafkaOffsetReader.

Issue Links

• links to: [Github] Pull Request #21134 (tdas)