[MAPREDUCE-3211] Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0-alpha
• Component/s: job submission, mrv2
• Labels: ShuffleHandler

Environment

Linux

• Hadoop Flags: Reviewed
• Release Note: Ephemeral port.
• Assignee: Prashant Sharma
• Reporter: Prashant Sharma
• Due: 19/Oct/11
• Created: 19/Oct/11 09:48
• Updated: 28/Sep/15 21:10
• Resolved: 20/Oct/11 05:28

Description

To reproduce the bug.

Use the following property in mapred-site.xml

<configuration>
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name> mapreduce.shuffle.port</name>
<value>0</value>
</property>
</configuration>

and following in yarn-site.xml

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Then try to start the yarn daemons.

And submit the job. (which would eventually fail).

Also observe NM logs. It says shuffle handler service started at port 0;