[SPARK-30446] Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 07/Jan/20 13:58
• Updated: 08/Jan/20 17:21
• Resolved: 08/Jan/20 17:21

Description

with accelerator aware scheduling SparkContext.checkResourcesPerTask

Tries to make sure that users have configured things properly and warn or error if not.

It doesn't properly handle all cases like warning if cpu resources are being wasted.  We should test this better and handle those. 

I fixed these in the stage level scheduling but not sure the timeline on getting that in so we may want to fix this separately as well.

Issue Links

• relates to: SPARK-24615 SPIP: Accelerator-aware task scheduling for Spark
• relates to: SPARK-30448 accelerator aware scheduling enforce cores as limiting resource