[SPARK-42834] Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 3.2.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: Li Ying
• Created: 17/Mar/23 09:44
• Updated: 18/Mar/23 03:15
• Resolved: 18/Mar/23 03:15

Description

Sometimes when run a SQL job with push based shuffle, exception occurs as below.  It seems that there’s no element in the bitmaps which stores merge chunk meta. 

Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?
 

Caused by: java.lang.ArithmeticException: / by zero
at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) 

related code:

def createChunkBlockInfosFromMetaResponse(
    shuffleId: Int,
    shuffleMergeId: Int,
    reduceId: Int,
    blockSize: Long,
    bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = {
  val approxChunkSize = blockSize / bitmaps.length
  val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]()
  for (i <- bitmaps.indices) {
    val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i)
    chunksMetaMap.put(blockChunkId, bitmaps(i))
    logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize")
    blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID))
  }
  blocksToFetch
} 

Issue Links

• is fixed by: SPARK-37675 Prevent overwriting of push shuffle merged files once the shuffle is finalized