[SPARK-44215] Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.3, 3.4.2, 3.5.0
• Component/s: Shuffle
• Assignee: Chandni Singh
• Reporter: Chandni Singh
• Created: 27/Jun/23 16:55
• Updated: 06/Jul/23 01:52
• Resolved: 04/Jul/23 23:48

Description

We still see instances of the server returning 0 numChunks in mergedMetaResponse which causes the executor to fail with ArithmeticException.

java.lang.ArithmeticException: / by zero
	at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:128)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:1047)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:90)
	at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)

Here the executor doesn't fallback to fetch un-merged blocks and this also doesn't result in a FetchFailure. So, the application fails.