Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
3.0.0
-
None
-
None
Description
with accelerator aware scheduling SparkContext.checkResourcesPerTask
Tries to make sure that users have configured things properly and warn or error if not.
It doesn't properly handle all cases like warning if cpu resources are being wasted. We should test this better and handle those.
I fixed these in the stage level scheduling but not sure the timeline on getting that in so we may want to fix this separately as well.
Attachments
Issue Links
- relates to
-
SPARK-24615 SPIP: Accelerator-aware task scheduling for Spark
- Resolved
-
SPARK-30448 accelerator aware scheduling enforce cores as limiting resource
- Resolved