[SPARK-44248] Kafka Source v2 should return preferred locations

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 29/Jun/23 18:21
• Updated: 29/Jun/23 21:50
• Resolved: 29/Jun/23 21:50

Description

DSv2 Kafka streaming source seems to miss setting the preferred location, which may destroy the purpose of cache for Kafka consumer (connection) & fetched data.

For DSv1, we have set the preferred location in RDD.

For DSv2, we should provide the info. in input partition, but we don't add the information into KafkaBatchInputPartition.