[SPARK-23717] Leverage docker support in Hadoop 3

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Mridul Muralidharan
• Created: 17/Mar/18 01:49
• Updated: 16/Mar/20 22:50

Description

The introduction of docker support in Apache Hadoop 3 can be leveraged by Apache Spark for resolving multiple long standing shortcoming - particularly related to package isolation.
It also allows for network isolation, where applicable, allowing for more sophisticated cluster configuration/customization.

This jira will track the various tasks for enhancing spark to leverage container support.

Issue Links

• is blocked by: SPARK-23718 Document using docker in host networking mode in hadoop 3
• is blocked by: SPARK-23719 Use correct hostname in non-host networking mode in hadoop 3 docker support
• is blocked by: SPARK-23720 Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support
• is blocked by: SPARK-23721 Enhance BlockManagerId to include container's underlying host machine hostname
• relates to: YARN-7994 Add support for network-alias in docker run for user defined networks