Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30446

Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.0.0
    • None
    • Spark Core
    • None

    Description

      with accelerator aware scheduling SparkContext.checkResourcesPerTask

      Tries to make sure that users have configured things properly and warn or error if not.

      It doesn't properly handle all cases like warning if cpu resources are being wasted.  We should test this better and handle those. 

      I fixed these in the stage level scheduling but not sure the timeline on getting that in so we may want to fix this separately as well.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: