Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8606

Exceptions in RDD.getPreferredLocations() and getPartitions() should not be able to crash DAGScheduler

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.3.0, 1.4.0, 1.5.0
    • Fix Version/s: 1.3.2, 1.4.1, 1.5.0
    • Component/s: Scheduler
    • Labels:
      None

      Description

      RDD.getPreferredLocations() and RDD.getPartitions() may throw exceptions but the DAGScheduler does not guard against this, leaving it vulnerable to crashing and stopping the SparkContext if exceptions occur there.

      We should fix this by adding more try blocks around these calls in DAGScheduler.

        Attachments

          Activity

            People

            • Assignee:
              joshrosen Josh Rosen
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: