[SPARK-28912] MatchError exception in CheckpointWriteHandler

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.3.2, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Assignee: Aleksandr Kashkirov
• Reporter: Aleksandr Kashkirov
• Created: 29/Aug/19 11:42
• Updated: 12/Dec/22 18:11
• Resolved: 07/Sep/19 00:56

Description

Setting checkpoint directory name to "checkpoint-" plus some digits (e.g. "checkpoint-01") results in the following error:

Exception in thread "pool-32-thread-1" scala.MatchError: 0523a434-0daa-4ea6-a050-c4eb3c557d8c (of class java.lang.String) 
 at org.apache.spark.streaming.Checkpoint$.org$apache$spark$streaming$Checkpoint$$sortFunc$1(Checkpoint.scala:121) 
 at org.apache.spark.streaming.Checkpoint$$anonfun$getCheckpointFiles$1.apply(Checkpoint.scala:132) 
 at org.apache.spark.streaming.Checkpoint$$anonfun$getCheckpointFiles$1.apply(Checkpoint.scala:132) 
 at scala.math.Ordering$$anon$9.compare(Ordering.scala:200) 
 at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) 
 at java.util.TimSort.sort(TimSort.java:234) 
 at java.util.Arrays.sort(Arrays.java:1438) 
 at scala.collection.SeqLike$class.sorted(SeqLike.scala:648) 
 at scala.collection.mutable.ArrayOps$ofRef.sorted(ArrayOps.scala:186) 
 at scala.collection.SeqLike$class.sortWith(SeqLike.scala:601) 
 at scala.collection.mutable.ArrayOps$ofRef.sortWith(ArrayOps.scala:186) 
 at org.apache.spark.streaming.Checkpoint$.getCheckpointFiles(Checkpoint.scala:132) 
 at org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler.run(Checkpoint.scala:262) 
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #25654
• links to: GitHub Pull Request #25719