Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26795

Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 2.3.0, 2.3.1, 2.3.2, 2.4.0
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      There is a parameter spark.maxRemoteBlockSizeFetchToMem, which means the remote block will be fetched to disk when size of the block is above this threshold in bytes.

      So during shuffle read phase, the managedBuffer which throw IOException may be a remote downloaded FileSegment and should be retried instead of throwFetchFailed directly.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                feiwang feiwang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: