[SPARK-27360] Standalone cluster mode support for GPU-aware scheduling

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Epic Link: GPU-aware Scheduling
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 04/Apr/19 00:30
• Updated: 27/Aug/19 14:01
• Resolved: 27/Aug/19 14:01

Description

Design and implement standalone manager support for GPU-aware scheduling:

1. static conf to describe resources
2. spark-submit to request resources
2. auto discovery of GPUs
3. executor process isolation

Issue Links

• is blocked by: SPARK-27024 Executor interface for cluster managers to support GPU resources

Sub-Tasks

• Design: Standalone supports GPU scheduling: Resolved
• Standalone worker can load resource conf and discover resources: Resolved
• Standalone master receives resource info from worker and allocate driver/executor properly: Resolved
• Standalone worker/master UI updates for Resource scheduling: Resolved
• Remove resource coordination support from Standalone: Resolved