Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30225

"Stream is corrupted at" exception on reading disk-spilled data of a shuffle operation

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.5, 3.0.0
    • Input/Output
    • None

    Description

      There is issues with spark.unsafe.sorter.spill.read.ahead.enabled in spark 2.4.0, which is introduced by https://issues.apache.org/jira/browse/SPARK-23366

       

      Workaround for this problem is to disable readahead of unsafe spill with following.
      --conf spark.unsafe.sorter.spill.read.ahead.enabled=false

       

      This issue can be reproduced on Spark 2.4.0 by following the steps in this comment of Jira SPARK-18105.

      https://issues.apache.org/jira/browse/SPARK-18105?focusedCommentId=16981461&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16981461

       

      Exception looks like below: 

      19/12/10 01:51:31 INFO sort.ShuffleExternalSorter: Thread 142 spilling sort data of 5.1 GB to disk (1  time so far)19/12/10 01:51:31 INFO sort.ShuffleExternalSorter: Thread 142 spilling sort data of 5.1 GB to disk (1  time so far)19/12/10 01:52:48 INFO sort.ShuffleExternalSorter: Thread 142 spilling sort data of 5.1 GB to disk (2  times so far)19/12/10 01:53:53 ERROR executor.Executor: Exception in task 6.0 in stage 0.0 (TID 6)java.io.IOException: Stream is corrupted at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:202) at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:228) at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157) at org.apache.spark.io.ReadAheadInputStream$1.run(ReadAheadInputStream.java:168) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)19/12/10 01:53:53 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 3319/12/10 01:53:53 INFO executor.Executor: Running task 8.1 in stage 0.0 (TID 33)19/12/10 01:54:00 INFO sort.UnsafeExternalSorter: Thread 142 spilling sort data of 3.3 GB to disk (0  time so far)19/12/10 01:54:30 INFO executor.Executor: Executor is trying to kill task 8.1 in stage 0.0 (TID 33), reason: Stage cancelled19/12/10 01:54:30 INFO executor.Executor: Executor killed task 8.1 in stage 0.0 (TID 33), reason: Stage cancelled19/12/10 01:54:52 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown

       

       

       

       

       

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vanzin Marcelo Masiero Vanzin
            mkempanna Mala Chikka Kempanna
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment