Technically the shuffle handler is a web service. If you want to change the default port I am fine with that, I am just curious what other web services do you expect to have running on the same node?

If we are changing the port, we might as well to change it to be in the same range as the rest of the defaults?

Didn't mean to imply that it's not a web service. Just that the nice thing about 8080 is that it's pretty easy for humans to remember, and thus a common place for apps to expose debug UIs and such. Since the shuffle handler port isn't really one that humans need to be visiting, and is one that other services are likely to use, it seems better to me not to use it.

We came across this when it had a conflict with Sqoop using the same port for something. Sqoop is going to move off of it as well.

Makes a lot of sense. +1 for the idea.

Submitted patch that changes the default port to 11000. The only rationale for that over any other number was that the other mapreduce ports, job history ports, are also in the 10000s, (19888 and 10020).

Verified that jobs still run on a single-node cluster.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12580614/MAPREDUCE-5036.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//console

This message is automatically generated.

+1 this looks fine to me.

we might want to pick a different default since oozie console also uses port 11000

ignore my last comment the default is actually 8080 in oozie.

I just committed this. Thanks Sandy.

Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Oy, I have just been informed that 11000 is the Oozie port.

After extensive googling and netstat'ing on a Hadoop cluster with a bunch of different Hadoop services, I've settled on 13562. Will post an addendum patch soon.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12585148/MAPREDUCE-5036-13562.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//console

This message is automatically generated.

+1 I'll commit this soon unless there are any objections to the new port number.

I just committed the fix. Thanks Sandy.

Integrated in Hadoop-trunk-Commit #3839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3839/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/)
MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

Sigh I totally missed this.

In reality we don't need this config at all. We should just bind to an ephemeral port. The AM and containers already have a way of learning all the meta information for aux services.

We probably still need the configuration since operators often like to control which ports daemons are using.

Sure, don't have issues leaving it around - may be useful in some cases. But the default can and should simply be zero.

The title doesnt reflect the change in the patch.

-  public static final int DEFAULT_SHUFFLE_PORT = 11000;
+  public static final int DEFAULT_SHUFFLE_PORT = 13562;

The title doesnt reflect the change in the patch.

The port was originally 8080, so I left the description the same so that it made sense for users reading the change log.

How about we do another update in this JIRA to change the default port to 0 as Vinod suggested?

Uploading a patch that changes to default to 0

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//console

This message is automatically generated.

+1. The mini cluster tests exercise this change, so no need for new tests.

I'll commit in the next day or so unless there are any further comments.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//console

This message is automatically generated.

Tom White / Vinod Kumar Vavilapalli - Do we still need this (0 default for shuffle handler port)? Or should we re-resolve as fixed in 2.4.0?

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
against trunk revision 05499b1.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console

This message is automatically generated.

I think changing it to an ephemeral port is good in general.

But the recent feature of NM-recovery (YARN-1336) can make this a problem.

• ShuffleHandler starts on port A
• MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer
• NM restarts and ShuffleHandler starts on port B now.
• Reducer fails to perform shuffle.

Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc Jason Lowe, Junping Du

Thanks Vinod, As it exists, is a static port config proving as a trouble
worthy enough that we would wanna support ephemeral ports?

Also, can this be done on a new JIRA, abandoning the follow-up patch that
this seems to be kept open for?

On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) <

–
Harsh J

Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts.

It's not just the aux-service ports but any other ports that clients use (i.e.: NM RPC port that AMs use to launch containers, http port for UI/logs, etc.) If these change during a restart then apps/UIs can fail. Yes, we could add the ability for this to be updated after a restart and broadcast to apps, but then apps have to broadcast that to their respective parts (e.g.: the MapReduce AM would have to inform reducers, hopefully in time to prevent them from declaring shuffle failure). IMHO all that complexity is simply not worth the trouble – it's far easier to just pick an unused port and reuse it on restart for any user-exposed services.

I'm OK with allowing people to configure ephemeral ports if they aren't using the restart functionality, but I think there are far better features/problems to tackle than trying to allow it when the restart feature is enabled.

Am marking this as resolved (since changes for this were committed into 2.4.0). I'll file up a new JIRA and link here for the ephemeral port support work, which appears to require more than resetting the port to 0 based on Vinod and Jason's comments above.

(Marking as Closed, as 2.4.0 has long since been released.)