[SPARK-45080] Kafka DSv2 streaming source implementation calls planInputPartitions 4 times per microbatch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 05/Sep/23 13:31
• Updated: 07/Sep/23 10:43
• Resolved: 07/Sep/23 10:43

Description

I was tracking through method calls for DSv2 streaming source, and figured out planInputPartitions is called 4 times per microbatch.

It turned out that multiple calls of planInputPartitions is due to `DataSourceV2ScanExecBase.supportsColumnar`, though it is called through `MicroBatchScanExec.inputPartitions` which is defined as lazy, hence shouldn't happen.

The behavior seems to be coupled with catalyst and very hard to figure out why, but with SPARK-44505, we can at least fix this per each data source.

Issue Links

• links to: [Github] Pull Request #42823 (HeartSaVioR)