Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38160

Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.3.0
    • None
    • SQL
    • None

    Description

      When we do shuffle on indeterminate expressions such as rand, and ShuffleFetchFailed happend, we may get incorrent result since it only retries failed map tasks.

      We try to fix this by retry all upstream map tasks in this situation.

      Attachments

        Activity

          People

            Unassigned Unassigned
            EdisonWang EdisonWang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: