[SPARK-45652] SPJ: Handle empty input partitions after dynamic filtering

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.4.2, 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 24/Oct/23 16:14
• Updated: 28/Mar/24 13:22
• Resolved: 26/Oct/23 15:52

Description

When the number of input partitions become 0 after dynamic filtering, in BatchScanExec, currently SPJ will fail with error:

java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions$lzycompute(BatchScanExec.scala:108)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions(BatchScanExec.scala:65)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD$lzycompute(BatchScanExec.scala:136)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD(BatchScanExec.scala:135)

This is because groupPartitions will return None for this case.

Issue Links

• links to: GitHub Pull Request #43531
• links to: GitHub Pull Request #43539
• links to: GitHub Pull Request #43540