[SPARK-16878] YARN - topology information

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Shubham Chopra
• Created: 03/Aug/16 14:28
• Updated: 17/May/20 18:13
• Resolved: 17/Mar/17 21:12

Description

Block replication strategies need topology information for ideal block placements. This information is available in resource managers and/or can be provided separately through scripts/services/classes, as in the case of HDFS. This jira focuses on enhancing spark-yarn package to suitably extract topology information from YARN.