Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20288

ImproveĀ BasicSchedulerIntegrationSuite "multi-stage job"

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      ShuffleId is determined before job submitted. But it's hard to predict stageId by shuffleId.
      Stage is created in DAGScheduler(
      https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L381), but the order is n
      ot determined.
      I added a log(println(s"Creating ShufflMapStage-$id on shuffle-${shuffleDep.shuffleId}")) after (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L331), when testing BasicSchedulerIntegrationSuite:"multi-stage job". It will print:
      Creating ShufflMapStage-0 on shuffle-0
      Creating ShufflMapStage-1 on shuffle-2
      Creating ShufflMapStage-2 on shuffle-1
      Creating ShufflMapStage-3 on shuffle-3
      or
      Creating ShufflMapStage-0 on shuffle-1
      Creating ShufflMapStage-1 on shuffle-3
      Creating ShufflMapStage-2 on shuffle-0
      Creating ShufflMapStage-3 on shuffle-2

      So It might be better to avoid generating the MapStatus by stageId.

        Attachments

          Activity

            People

            • Assignee:
              jinxing6042@126.com jin xing
              Reporter:
              jinxing6042@126.com jin xing
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: