[SPARK-27823] Add an abstraction layer for accelerator resource handling to avoid manipulating raw confs

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Target Version/s: 3.0.0
• Epic Link: GPU-aware Scheduling
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 23/May/19 18:49
• Updated: 22/Jan/20 22:06
• Resolved: 19/Jun/19 00:21

Description

In SPARK-27488, we extract resource requests and allocation by parsing raw Spark confs. This hurts readability because we didn't have the abstraction at resource level. After we merge the core changes, we should do a refactoring and make the code more readable.

See https://github.com/apache/spark/pull/24615#issuecomment-494580663.

Issue Links

• is blocked by: SPARK-27366 Spark scheduler internal changes to support GPU scheduling
• is blocked by: SPARK-27488 Driver interface to support GPU resources
• links to: GitHub Pull Request #24821
• links to: GitHub Pull Request #24856
• links to: GitHub Pull Request #24954