[SPARK-23720] Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Mridul Muralidharan
• Created: 17/Mar/18 02:11
• Updated: 16/Mar/20 22:53

Description

In current external shuffle service integration, hostname of the executor and the shuffle service is the same while the port's are different (shuffle service port vs block manager port).
When running in non-host networking mode under docker, in yarn, the shuffle service runs on the NM_HOST while the docker container run's under its own (ephemeral and generated) hostname.

We should make use of the container's host machine's hostname for shuffle service and not the container hostname, when external shuffle is enabled.

Issue Links

• blocks: SPARK-23717 Leverage docker support in Hadoop 3
• is blocked by: SPARK-23719 Use correct hostname in non-host networking mode in hadoop 3 docker support