[SPARK-40459] recoverDiskStore should not stop by existing recomputed files

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.3, 3.3.2
• Fix Version/s: 3.3.1, 3.2.3, 3.4.0
• Component/s: Kubernetes
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 16/Sep/22 00:20
• Updated: 14/Dec/22 05:46
• Resolved: 16/Sep/22 01:04

Description

org.apache.commons.io.FileExistsException: File element in parameter 'null' already exists: '...'
	at org.apache.commons.io.FileUtils.requireAbsent(FileUtils.java:2587)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2305)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2283)
	at org.apache.spark.storage.DiskStore.moveFileToBlock(DiskStore.scala:150)
	at org.apache.spark.storage.BlockManager$TempFileBasedBlockStoreUpdater.saveToDiskStore(BlockManager.scala:487)
	at org.apache.spark.storage.BlockManager$BlockStoreUpdater.$anonfun$save$1(BlockManager.scala:407)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1445)
	at org.apache.spark.storage.BlockManager$BlockStoreUpdater.save(BlockManager.scala:380)
	at org.apache.spark.storage.BlockManager$TempFileBasedBlockStoreUpdater.save(BlockManager.scala:490)
	at org.apache.spark.shuffle.KubernetesLocalDiskShuffleExecutorComponents$.$anonfun$recoverDiskStore$14(KubernetesLocalDiskShuffleExecutorComponents.scala:95)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.apache.spark.shuffle.KubernetesLocalDiskShuffleExecutorComponents$.recoverDiskStore(KubernetesLocalDiskShuffleExecutorComponents.scala:91) 

Issue Links

• links to: [Github] Pull Request #37903 (dongjoon-hyun)
• links to: [Github] Pull Request #37903 (dongjoon-hyun)