Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28912

MatchError exception in CheckpointWriteHandler

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 2.3.2, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4
    • Fix Version/s: 2.4.5, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Setting checkpoint directory name to "checkpoint-" plus some digits (e.g. "checkpoint-01") results in the following error:

      Exception in thread "pool-32-thread-1" scala.MatchError: 0523a434-0daa-4ea6-a050-c4eb3c557d8c (of class java.lang.String) 
       at org.apache.spark.streaming.Checkpoint$.org$apache$spark$streaming$Checkpoint$$sortFunc$1(Checkpoint.scala:121) 
       at org.apache.spark.streaming.Checkpoint$$anonfun$getCheckpointFiles$1.apply(Checkpoint.scala:132) 
       at org.apache.spark.streaming.Checkpoint$$anonfun$getCheckpointFiles$1.apply(Checkpoint.scala:132) 
       at scala.math.Ordering$$anon$9.compare(Ordering.scala:200) 
       at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) 
       at java.util.TimSort.sort(TimSort.java:234) 
       at java.util.Arrays.sort(Arrays.java:1438) 
       at scala.collection.SeqLike$class.sorted(SeqLike.scala:648) 
       at scala.collection.mutable.ArrayOps$ofRef.sorted(ArrayOps.scala:186) 
       at scala.collection.SeqLike$class.sortWith(SeqLike.scala:601) 
       at scala.collection.mutable.ArrayOps$ofRef.sortWith(ArrayOps.scala:186) 
       at org.apache.spark.streaming.Checkpoint$.getCheckpointFiles(Checkpoint.scala:132) 
       at org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler.run(Checkpoint.scala:262) 
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
       at java.lang.Thread.run(Thread.java:748)

        Attachments

          Activity

            People

            • Assignee:
              avk1 Aleksandr Kashkirov
              Reporter:
              avk1 Aleksandr Kashkirov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: