Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5036

Default shuffle handler port should not be 8080

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts.

      1. MAPREDUCE-5036.patch
        2 kB
        Sandy Ryza
      2. MAPREDUCE-5036-13562.patch
        2 kB
        Sandy Ryza
      3. MAPREDUCE-5036-2.patch
        2 kB
        Sandy Ryza

        Issue Links

          Activity

          Hide
          Robert Joseph Evans added a comment -

          Technically the shuffle handler is a web service. If you want to change the default port I am fine with that, I am just curious what other web services do you expect to have running on the same node?

          Show
          Robert Joseph Evans added a comment - Technically the shuffle handler is a web service. If you want to change the default port I am fine with that, I am just curious what other web services do you expect to have running on the same node?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          If we are changing the port, we might as well to change it to be in the same range as the rest of the defaults?

          Show
          Karthik Kambatla (Inactive) added a comment - If we are changing the port, we might as well to change it to be in the same range as the rest of the defaults?
          Hide
          Sandy Ryza added a comment -

          Didn't mean to imply that it's not a web service. Just that the nice thing about 8080 is that it's pretty easy for humans to remember, and thus a common place for apps to expose debug UIs and such. Since the shuffle handler port isn't really one that humans need to be visiting, and is one that other services are likely to use, it seems better to me not to use it.

          We came across this when it had a conflict with Sqoop using the same port for something. Sqoop is going to move off of it as well.

          Show
          Sandy Ryza added a comment - Didn't mean to imply that it's not a web service. Just that the nice thing about 8080 is that it's pretty easy for humans to remember, and thus a common place for apps to expose debug UIs and such. Since the shuffle handler port isn't really one that humans need to be visiting, and is one that other services are likely to use, it seems better to me not to use it. We came across this when it had a conflict with Sqoop using the same port for something. Sqoop is going to move off of it as well.
          Hide
          Robert Joseph Evans added a comment -

          Makes a lot of sense. +1 for the idea.

          Show
          Robert Joseph Evans added a comment - Makes a lot of sense. +1 for the idea.
          Hide
          Sandy Ryza added a comment -

          Submitted patch that changes the default port to 11000. The only rationale for that over any other number was that the other mapreduce ports, job history ports, are also in the 10000s, (19888 and 10020).

          Verified that jobs still run on a single-node cluster.

          Show
          Sandy Ryza added a comment - Submitted patch that changes the default port to 11000. The only rationale for that over any other number was that the other mapreduce ports, job history ports, are also in the 10000s, (19888 and 10020). Verified that jobs still run on a single-node cluster.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580614/MAPREDUCE-5036.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580614/MAPREDUCE-5036.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3547//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1 this looks fine to me.

          Show
          Tom White added a comment - +1 this looks fine to me.
          Hide
          Arpit Gupta added a comment -

          we might want to pick a different default since oozie console also uses port 11000

          Show
          Arpit Gupta added a comment - we might want to pick a different default since oozie console also uses port 11000
          Hide
          Arpit Gupta added a comment -

          ignore my last comment the default is actually 8080 in oozie.

          Show
          Arpit Gupta added a comment - ignore my last comment the default is actually 8080 in oozie.
          Hide
          Tom White added a comment -

          I just committed this. Thanks Sandy.

          Show
          Tom White added a comment - I just committed this. Thanks Sandy.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

          Result = FAILURE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1478422) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478422 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Sandy Ryza added a comment -

          Oy, I have just been informed that 11000 is the Oozie port.

          Show
          Sandy Ryza added a comment - Oy, I have just been informed that 11000 is the Oozie port.
          Hide
          Sandy Ryza added a comment -

          After extensive googling and netstat'ing on a Hadoop cluster with a bunch of different Hadoop services, I've settled on 13562. Will post an addendum patch soon.

          Show
          Sandy Ryza added a comment - After extensive googling and netstat'ing on a Hadoop cluster with a bunch of different Hadoop services, I've settled on 13562. Will post an addendum patch soon.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585148/MAPREDUCE-5036-13562.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585148/MAPREDUCE-5036-13562.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3684//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1 I'll commit this soon unless there are any objections to the new port number.

          Show
          Tom White added a comment - +1 I'll commit this soon unless there are any objections to the new port number.
          Hide
          Tom White added a comment -

          I just committed the fix. Thanks Sandy.

          Show
          Tom White added a comment - I just committed the fix. Thanks Sandy.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3839/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3839/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

          Result = FAILURE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

          Result = FAILURE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/)
          MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/ ) MAPREDUCE-5036 . Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489060 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Sigh I totally missed this.

          In reality we don't need this config at all. We should just bind to an ephemeral port. The AM and containers already have a way of learning all the meta information for aux services.

          Show
          Vinod Kumar Vavilapalli added a comment - Sigh I totally missed this. In reality we don't need this config at all. We should just bind to an ephemeral port. The AM and containers already have a way of learning all the meta information for aux services.
          Hide
          Tom White added a comment -

          We probably still need the configuration since operators often like to control which ports daemons are using.

          Show
          Tom White added a comment - We probably still need the configuration since operators often like to control which ports daemons are using.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Sure, don't have issues leaving it around - may be useful in some cases. But the default can and should simply be zero.

          Show
          Vinod Kumar Vavilapalli added a comment - Sure, don't have issues leaving it around - may be useful in some cases. But the default can and should simply be zero.
          Hide
          Bikas Saha added a comment -

          The title doesnt reflect the change in the patch.

          -  public static final int DEFAULT_SHUFFLE_PORT = 11000;
          +  public static final int DEFAULT_SHUFFLE_PORT = 13562;
          
          Show
          Bikas Saha added a comment - The title doesnt reflect the change in the patch. - public static final int DEFAULT_SHUFFLE_PORT = 11000; + public static final int DEFAULT_SHUFFLE_PORT = 13562;
          Hide
          Tom White added a comment -

          The title doesnt reflect the change in the patch.

          The port was originally 8080, so I left the description the same so that it made sense for users reading the change log.

          How about we do another update in this JIRA to change the default port to 0 as Vinod suggested?

          Show
          Tom White added a comment - The title doesnt reflect the change in the patch. The port was originally 8080, so I left the description the same so that it made sense for users reading the change log. How about we do another update in this JIRA to change the default port to 0 as Vinod suggested?
          Hide
          Sandy Ryza added a comment -

          Uploading a patch that changes to default to 0

          Show
          Sandy Ryza added a comment - Uploading a patch that changes to default to 0
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3761//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1. The mini cluster tests exercise this change, so no need for new tests.

          I'll commit in the next day or so unless there are any further comments.

          Show
          Tom White added a comment - +1. The mini cluster tests exercise this change, so no need for new tests. I'll commit in the next day or so unless there are any further comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4341//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Tom White / Vinod Kumar Vavilapalli - Do we still need this (0 default for shuffle handler port)? Or should we re-resolve as fixed in 2.4.0?

          Show
          Harsh J added a comment - Tom White / Vinod Kumar Vavilapalli - Do we still need this (0 default for shuffle handler port)? Or should we re-resolve as fixed in 2.4.0?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch
          against trunk revision 05499b1.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision 05499b1. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I think changing it to an ephemeral port is good in general.

          But the recent feature of NM-recovery (YARN-1336) can make this a problem.

          • ShuffleHandler starts on port A
          • MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer
          • NM restarts and ShuffleHandler starts on port B now.
          • Reducer fails to perform shuffle.

          Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc Jason Lowe, Junping Du

          Show
          Vinod Kumar Vavilapalli added a comment - I think changing it to an ephemeral port is good in general. But the recent feature of NM-recovery ( YARN-1336 ) can make this a problem. ShuffleHandler starts on port A MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer NM restarts and ShuffleHandler starts on port B now. Reducer fails to perform shuffle. Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc Jason Lowe , Junping Du
          Hide
          Harsh J added a comment -

          Thanks Vinod, As it exists, is a static port config proving as a trouble
          worthy enough that we would wanna support ephemeral ports?

          Also, can this be done on a new JIRA, abandoning the follow-up patch that
          this seems to be kept open for?

          On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) <


          Harsh J

          Show
          Harsh J added a comment - Thanks Vinod, As it exists, is a static port config proving as a trouble worthy enough that we would wanna support ephemeral ports? Also, can this be done on a new JIRA, abandoning the follow-up patch that this seems to be kept open for? On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) < – Harsh J
          Hide
          Jason Lowe added a comment -

          Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts.

          It's not just the aux-service ports but any other ports that clients use (i.e.: NM RPC port that AMs use to launch containers, http port for UI/logs, etc.) If these change during a restart then apps/UIs can fail. Yes, we could add the ability for this to be updated after a restart and broadcast to apps, but then apps have to broadcast that to their respective parts (e.g.: the MapReduce AM would have to inform reducers, hopefully in time to prevent them from declaring shuffle failure). IMHO all that complexity is simply not worth the trouble – it's far easier to just pick an unused port and reuse it on restart for any user-exposed services.

          I'm OK with allowing people to configure ephemeral ports if they aren't using the restart functionality, but I think there are far better features/problems to tackle than trying to allow it when the restart feature is enabled.

          Show
          Jason Lowe added a comment - Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. It's not just the aux-service ports but any other ports that clients use (i.e.: NM RPC port that AMs use to launch containers, http port for UI/logs, etc.) If these change during a restart then apps/UIs can fail. Yes, we could add the ability for this to be updated after a restart and broadcast to apps, but then apps have to broadcast that to their respective parts (e.g.: the MapReduce AM would have to inform reducers, hopefully in time to prevent them from declaring shuffle failure). IMHO all that complexity is simply not worth the trouble – it's far easier to just pick an unused port and reuse it on restart for any user-exposed services. I'm OK with allowing people to configure ephemeral ports if they aren't using the restart functionality, but I think there are far better features/problems to tackle than trying to allow it when the restart feature is enabled.
          Hide
          Harsh J added a comment -

          Am marking this as resolved (since changes for this were committed into 2.4.0). I'll file up a new JIRA and link here for the ephemeral port support work, which appears to require more than resetting the port to 0 based on Vinod and Jason's comments above.

          Show
          Harsh J added a comment - Am marking this as resolved (since changes for this were committed into 2.4.0). I'll file up a new JIRA and link here for the ephemeral port support work, which appears to require more than resetting the port to 0 based on Vinod and Jason's comments above.
          Hide
          Harsh J added a comment -

          (Marking as Closed, as 2.4.0 has long since been released.)

          Show
          Harsh J added a comment - (Marking as Closed, as 2.4.0 has long since been released.)

            People

            • Assignee:
              Sandy Ryza
              Reporter:
              Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development