Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5036

Default shuffle handler port should not be 8080

        Details

        • Type: Improvement
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 2.0.3-alpha
        • Fix Version/s: 2.4.0
        • Component/s: None
        • Labels:
          None
        • Hadoop Flags:
          Incompatible change, Reviewed

          Description

          The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts.

          1. Text File
            MAPREDUCE-5036.patch
            2 kB
            Sandy Ryza
          2. Text File
            MAPREDUCE-5036-13562.patch
            2 kB
            Sandy Ryza
          3. Text File
            MAPREDUCE-5036-2.patch
            2 kB
            Sandy Ryza

            Issue Links

            incorporates

            New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6325 Support use of ephemeral client ports for shuffle/etc.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              qwertymaniac Harsh J added a comment -

              (Marking as Closed, as 2.4.0 has long since been released.)

              Show
              qwertymaniac Harsh J added a comment - (Marking as Closed, as 2.4.0 has long since been released.)
              Hide
              Permalink
              qwertymaniac Harsh J added a comment -

              Am marking this as resolved (since changes for this were committed into 2.4.0). I'll file up a new JIRA and link here for the ephemeral port support work, which appears to require more than resetting the port to 0 based on Vinod and Jason's comments above.

              Show
              qwertymaniac Harsh J added a comment - Am marking this as resolved (since changes for this were committed into 2.4.0). I'll file up a new JIRA and link here for the ephemeral port support work, which appears to require more than resetting the port to 0 based on Vinod and Jason's comments above.
              Hide
              Permalink
              jlowe Jason Lowe added a comment -

              Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts.

              It's not just the aux-service ports but any other ports that clients use (i.e.: NM RPC port that AMs use to launch containers, http port for UI/logs, etc.) If these change during a restart then apps/UIs can fail. Yes, we could add the ability for this to be updated after a restart and broadcast to apps, but then apps have to broadcast that to their respective parts (e.g.: the MapReduce AM would have to inform reducers, hopefully in time to prevent them from declaring shuffle failure). IMHO all that complexity is simply not worth the trouble – it's far easier to just pick an unused port and reuse it on restart for any user-exposed services.

              I'm OK with allowing people to configure ephemeral ports if they aren't using the restart functionality, but I think there are far better features/problems to tackle than trying to allow it when the restart feature is enabled.

              Show
              jlowe Jason Lowe added a comment - Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. It's not just the aux-service ports but any other ports that clients use (i.e.: NM RPC port that AMs use to launch containers, http port for UI/logs, etc.) If these change during a restart then apps/UIs can fail. Yes, we could add the ability for this to be updated after a restart and broadcast to apps, but then apps have to broadcast that to their respective parts (e.g.: the MapReduce AM would have to inform reducers, hopefully in time to prevent them from declaring shuffle failure). IMHO all that complexity is simply not worth the trouble – it's far easier to just pick an unused port and reuse it on restart for any user-exposed services. I'm OK with allowing people to configure ephemeral ports if they aren't using the restart functionality, but I think there are far better features/problems to tackle than trying to allow it when the restart feature is enabled.
              Hide
              Permalink
              qwertymaniac Harsh J added a comment -

              Thanks Vinod, As it exists, is a static port config proving as a trouble
              worthy enough that we would wanna support ephemeral ports?

              Also, can this be done on a new JIRA, abandoning the follow-up patch that
              this seems to be kept open for?

              On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) <

              –
              Harsh J

              Show
              qwertymaniac Harsh J added a comment - Thanks Vinod, As it exists, is a static port config proving as a trouble worthy enough that we would wanna support ephemeral ports? Also, can this be done on a new JIRA, abandoning the follow-up patch that this seems to be kept open for? On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) < – Harsh J
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              I think changing it to an ephemeral port is good in general.

              But the recent feature of NM-recovery (YARN-1336) can make this a problem.

              • ShuffleHandler starts on port A
              • MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer
              • NM restarts and ShuffleHandler starts on port B now.
              • Reducer fails to perform shuffle.

              Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc Jason Lowe, Junping Du

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - I think changing it to an ephemeral port is good in general. But the recent feature of NM-recovery ( YARN-1336 ) can make this a problem. ShuffleHandler starts on port A MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer NM restarts and ShuffleHandler starts on port B now. Reducer fails to perform shuffle. Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc Jason Lowe , Junping Du
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
              against trunk revision 05499b1.

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision 05499b1. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console This message is automatically generated.
              Hide
              Permalink
              qwertymaniac Harsh J added a comment -

              Tom White / Vinod Kumar Vavilapalli - Do we still need this (0 default for shuffle handler port)? Or should we re-resolve as fixed in 2.4.0?

              Show
              qwertymaniac Harsh J added a comment - Tom White / Vinod Kumar Vavilapalli - Do we still need this (0 default for shuffle handler port)? Or should we re-resolve as fixed in 2.4.0?
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//console This message is automatically generated.
              Hide
              Permalink
              tomwhite Tom White added a comment -

              +1. The mini cluster tests exercise this change, so no need for new tests.

              I'll commit in the next day or so unless there are any further comments.

              Show
              tomwhite Tom White added a comment - +1. The mini cluster tests exercise this change, so no need for new tests. I'll commit in the next day or so unless there are any further comments.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//console This message is automatically generated.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              Uploading a patch that changes to default to 0

              Show
              sandyr Sandy Ryza added a comment - Uploading a patch that changes to default to 0
              Hide
              Permalink
              tomwhite Tom White added a comment -

              The title doesnt reflect the change in the patch.

              The port was originally 8080, so I left the description the same so that it made sense for users reading the change log.

              How about we do another update in this JIRA to change the default port to 0 as Vinod suggested?

              Show
              tomwhite Tom White added a comment - The title doesnt reflect the change in the patch. The port was originally 8080, so I left the description the same so that it made sense for users reading the change log. How about we do another update in this JIRA to change the default port to 0 as Vinod suggested?
              Hide
              Permalink
              bikassaha Bikas Saha added a comment -

              The title doesnt reflect the change in the patch.

              -  public static final int DEFAULT_SHUFFLE_PORT = 11000;
+  public static final int DEFAULT_SHUFFLE_PORT = 13562;
              Show
              bikassaha Bikas Saha added a comment - The title doesnt reflect the change in the patch. - public static final int DEFAULT_SHUFFLE_PORT = 11000; + public static final int DEFAULT_SHUFFLE_PORT = 13562;
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              Sure, don't have issues leaving it around - may be useful in some cases. But the default can and should simply be zero.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - Sure, don't have issues leaving it around - may be useful in some cases. But the default can and should simply be zero.
              Hide
              Permalink
              tomwhite Tom White added a comment -

              We probably still need the configuration since operators often like to control which ports daemons are using.

              Show
              tomwhite Tom White added a comment - We probably still need the configuration since operators often like to control which ports daemons are using.
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              Sigh I totally missed this.

              In reality we don't need this config at all. We should just bind to an ephemeral port. The AM and containers already have a way of learning all the meta information for aux services.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - Sigh I totally missed this. In reality we don't need this config at all. We should just bind to an ephemeral port. The AM and containers already have a way of learning all the meta information for aux services.
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

              Result = SUCCESS
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

              Result = FAILURE
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

              Result = FAILURE
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3839/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

              Result = SUCCESS
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3839/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              tomwhite Tom White added a comment -

              I just committed the fix. Thanks Sandy.

              Show
              tomwhite Tom White added a comment - I just committed the fix. Thanks Sandy.
              Hide
              Permalink
              tomwhite Tom White added a comment -

              +1 I'll commit this soon unless there are any objections to the new port number.

              Show
              tomwhite Tom White added a comment - +1 I'll commit this soon unless there are any objections to the new port number.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12585148/MAPREDUCE-5036-13562.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585148/MAPREDUCE-5036-13562.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//console This message is automatically generated.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              After extensive googling and netstat'ing on a Hadoop cluster with a bunch of different Hadoop services, I've settled on 13562. Will post an addendum patch soon.

              Show
              sandyr Sandy Ryza added a comment - After extensive googling and netstat'ing on a Hadoop cluster with a bunch of different Hadoop services, I've settled on 13562. Will post an addendum patch soon.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              Oy, I have just been informed that 11000 is the Oozie port.

              Show
              sandyr Sandy Ryza added a comment - Oy, I have just been informed that 11000 is the Oozie port.
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

              Result = SUCCESS
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

              Result = FAILURE
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

              Result = SUCCESS
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/)
              MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

              Result = SUCCESS
              tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
              Hide
              Permalink
              tomwhite Tom White added a comment -

              I just committed this. Thanks Sandy.

              Show
              tomwhite Tom White added a comment - I just committed this. Thanks Sandy.
              Hide
              Permalink
              arpitgupta Arpit Gupta added a comment -

              ignore my last comment the default is actually 8080 in oozie.

              Show
              arpitgupta Arpit Gupta added a comment - ignore my last comment the default is actually 8080 in oozie.
              Hide
              Permalink
              arpitgupta Arpit Gupta added a comment -

              we might want to pick a different default since oozie console also uses port 11000

              Show
              arpitgupta Arpit Gupta added a comment - we might want to pick a different default since oozie console also uses port 11000
              Hide
              Permalink
              tomwhite Tom White added a comment -

              +1 this looks fine to me.

              Show
              tomwhite Tom White added a comment - +1 this looks fine to me.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12580614/MAPREDUCE-5036.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580614/MAPREDUCE-5036.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//console This message is automatically generated.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              Submitted patch that changes the default port to 11000. The only rationale for that over any other number was that the other mapreduce ports, job history ports, are also in the 10000s, (19888 and 10020).

              Verified that jobs still run on a single-node cluster.

              Show
              sandyr Sandy Ryza added a comment - Submitted patch that changes the default port to 11000. The only rationale for that over any other number was that the other mapreduce ports, job history ports, are also in the 10000s, (19888 and 10020). Verified that jobs still run on a single-node cluster.
              Hide
              Permalink
              revans2 Robert Joseph Evans added a comment -

              Makes a lot of sense. +1 for the idea.

              Show
              revans2 Robert Joseph Evans added a comment - Makes a lot of sense. +1 for the idea.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              Didn't mean to imply that it's not a web service. Just that the nice thing about 8080 is that it's pretty easy for humans to remember, and thus a common place for apps to expose debug UIs and such. Since the shuffle handler port isn't really one that humans need to be visiting, and is one that other services are likely to use, it seems better to me not to use it.

              We came across this when it had a conflict with Sqoop using the same port for something. Sqoop is going to move off of it as well.

              Show
              sandyr Sandy Ryza added a comment - Didn't mean to imply that it's not a web service. Just that the nice thing about 8080 is that it's pretty easy for humans to remember, and thus a common place for apps to expose debug UIs and such. Since the shuffle handler port isn't really one that humans need to be visiting, and is one that other services are likely to use, it seems better to me not to use it. We came across this when it had a conflict with Sqoop using the same port for something. Sqoop is going to move off of it as well.
              Hide
              Permalink
              kkambatl Karthik Kambatla (Inactive) added a comment -

              If we are changing the port, we might as well to change it to be in the same range as the rest of the defaults?

              Show
              kkambatl Karthik Kambatla (Inactive) added a comment - If we are changing the port, we might as well to change it to be in the same range as the rest of the defaults?
              Hide
              Permalink
              revans2 Robert Joseph Evans added a comment -

              Technically the shuffle handler is a web service. If you want to change the default port I am fine with that, I am just curious what other web services do you expect to have running on the same node?

              Show
              revans2 Robert Joseph Evans added a comment - Technically the shuffle handler is a web service. If you want to change the default port I am fine with that, I am just curious what other web services do you expect to have running on the same node?

                People

                • Assignee:
                  sandyr Sandy Ryza
                  Reporter:
                  sandyr Sandy Ryza
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  15 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development