Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3532

When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.1
    • Component/s: mrv2, nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modified NM to report correct http address when an ephemeral web port is configured.

      Description

      I tried following -:
      yarn.nodemanager.address=0.0.0.0:0
      yarn.nodemanager.webapp.address=0.0.0.0:0
      yarn.nodemanager.localizer.address=0.0.0.0:0
      mapreduce.shuffle.port=0

      When 0 is provided as number in yarn.nodemanager.webapp.address.
      NM instantiate WebServer as 0 piort e.g.

      2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
      

      After that WebServer pick up some random port e.g.

      2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
      2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
      2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
      2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
      

      And NM WebServer responds correctly but
      RM's cluster/Nodes page shows the following -:

      /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
      

      Whereas NM:0 is not clickable.
      Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
      But verified that MR job runs successfully with random.

      1. MAPREDUCE-3532-1.patch
        4 kB
        Bhallamudi Venkata Siva Kamesh
      2. MAPREDUCE-3532.patch
        1 kB
        Bhallamudi Venkata Siva Kamesh

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        23d 12h 45m 1 Bhallamudi Venkata Siva Kamesh 04/Jan/12 18:45
        Patch Available Patch Available Open Open
        9d 3h 22m 1 Vinod Kumar Vavilapalli 13/Jan/12 22:08
        Open Open Resolved Resolved
        15m 1s 1 Vinod Kumar Vavilapalli 13/Jan/12 22:23
        Resolved Resolved Closed Closed
        51d 4h 26m 1 Arun C Murthy 05/Mar/12 02:49
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #958 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #958 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
        svn merge --ignore-ancestry -c 1231342 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. svn merge --ignore-ancestry -c 1231342 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #138 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
        svn merge --ignore-ancestry -c 1231342 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #138 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. svn merge --ignore-ancestry -c 1231342 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #925 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #925 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1559/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1559/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #387 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/387/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
        svn merge --ignore-ancestry -c 1231342 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #387 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/387/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. svn merge --ignore-ancestry -c 1231342 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1614 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1614/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1614 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1614/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #365 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/365/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
        svn merge --ignore-ancestry -c 1231342 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #365 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/365/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. svn merge --ignore-ancestry -c 1231342 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1541 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1541/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1541 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1541/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231342 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #375 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/375/)
        MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
        svn merge --ignore-ancestry -c 1231342 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #375 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/375/ ) MAPREDUCE-3532 . Modified NM to report correct http address when an ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh. svn merge --ignore-ancestry -c 1231342 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231344 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Modified NM to report correct http address when an ephemeral web port is configured.
        Resolution Fixed [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk and branch-0.23. Thanks Kamesh!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk and branch-0.23. Thanks Kamesh!
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Fix Version/s 0.23.1 [ 12318883 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I looked through the patch. Looks good. +1.

        Show
        Vinod Kumar Vavilapalli added a comment - I looked through the patch. Looks good. +1.
        Vinod Kumar Vavilapalli made changes -
        Assignee Bhallamudi Venkata Siva Kamesh [ kamesh ]
        Siddharth Seth made changes -
        Component/s mrv2 [ 12314301 ]
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        TestContainersMonitor is passing in my env before and after code changes.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - TestContainersMonitor is passing in my env before and after code changes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509439/MAPREDUCE-3532-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1535//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1535//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509439/MAPREDUCE-3532-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1535//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1535//console This message is automatically generated.
        Bhallamudi Venkata Siva Kamesh made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Bhallamudi Venkata Siva Kamesh made changes -
        Attachment MAPREDUCE-3532-1.patch [ 12509439 ]
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Attaching patch with Testcases. Pls review

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Attaching patch with Testcases. Pls review
        Bhallamudi Venkata Siva Kamesh made changes -
        Attachment MAPREDUCE-3532.patch [ 12508932 ]
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Updating the NM_WEBAPP_ADDRESS with NM's webserver http port. Attached the patch for the same. Please review.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Updating the NM_WEBAPP_ADDRESS with NM's webserver http port. Attached the patch for the same. Please review.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        When we set 0 as port for NM_WEBAPP_ADDRESS, this.httpPort will be assigned the same value. While sending registeration requestion to the RM, NM sends the same http port as part of its request. In this case http port happens 0.

        NodeStatusUpdaterImpl.java
            InetSocketAddress httpBindAddress =
              NetUtils.createSocketAddr(httpBindAddressStr,
                YarnConfiguration.DEFAULT_NM_WEBAPP_PORT,
                YarnConfiguration.NM_WEBAPP_ADDRESS);
            try {
              //      this.hostName = InetAddress.getLocalHost().getCanonicalHostName();
              this.httpPort = httpBindAddress.getPort();
        
        Show
        Bhallamudi Venkata Siva Kamesh added a comment - When we set 0 as port for NM_WEBAPP_ADDRESS , this.httpPort will be assigned the same value. While sending registeration requestion to the RM, NM sends the same http port as part of its request. In this case http port happens 0. NodeStatusUpdaterImpl.java InetSocketAddress httpBindAddress = NetUtils.createSocketAddr(httpBindAddressStr, YarnConfiguration.DEFAULT_NM_WEBAPP_PORT, YarnConfiguration.NM_WEBAPP_ADDRESS); try { // this .hostName = InetAddress.getLocalHost().getCanonicalHostName(); this .httpPort = httpBindAddress.getPort();
        Mahadev konar made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Description I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 port e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
        {code}
        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 piort e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
        {code}
        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        Karam Singh made changes -
        Description I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 piort e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
        {code}
        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 port e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
        {code}
        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        Karam Singh made changes -
        Summary When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port ot RM When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
        Karam Singh made changes -
        Field Original Value New Value
        Description I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 piort e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB

        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        I tried following -:
        yarn.nodemanager.address=0.0.0.0:0
        yarn.nodemanager.webapp.address=0.0.0.0:0
        yarn.nodemanager.localizer.address=0.0.0.0:0
        mapreduce.shuffle.port=0

        When 0 is provided as number in yarn.nodemanager.webapp.address.
        NM instantiate WebServer as 0 piort e.g.
        {code}
        2011-12-08 11:33:02,467 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:0
        {code}

        After that WebServer pick up some random port e.g.
        {code}
        2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 36272
        2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
        2011-12-08 11:33:02,831 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:36272
        2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 36272
        {code}

        And NM WebServer responds correctly but
         RM's cluster/Nodes page shows the following -:
        {code}
        /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
        {code}
        Whereas NM:0 is not clickable.
        Seems even NM's webserver pick random port but it never gets updated and so NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
        But verified that MR job runs successfully with random.
        Karam Singh created issue -

          People

          • Assignee:
            Bhallamudi Venkata Siva Kamesh
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development