[SPARK-44012] KafkaDataConsumer to log some stats

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 09/Jun/23 03:27
• Updated: 20/Jun/23 22:34
• Resolved: 20/Jun/23 22:34

Description

For each task, Kafka source should report fraction of time spent in KafkaConsumer to fetch records. It should also report overall read bandwidth (bytes or records read / time spent fetching).

This will be useful in verifying if fetching is the bottleneck.