Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2990

Health Report on Resource Manager UI is null if the NM's are all healthy.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      The web UI on the RM for the link Nodes shows that Health-report as null when the NM is healthy.

      This is a simple fix where in we can check for null in NodesPage.java and put something meaningful instead of null.
      NodesPage.java:

      
      render(..)
      
      td((health.getHealthReport() == null) ?"REPORT HEALTHY": health.getHealthReport());
      
      

      Or something like that.

      1. MAPREDUCE-2990(1).patch
        1 kB
        Subroto Sanyal
      2. MAPREDUCE-2990.1.patch
        5 kB
        Subroto Sanyal
      3. MAPREDUCE-2990.patch
        5 kB
        Arun C Murthy

        Activity

        Hide
        ramkrishna.s.vasudevan added a comment -

        Attaching the patch for UI change....

        Show
        ramkrishna.s.vasudevan added a comment - Attaching the patch for UI change....
        Hide
        Subroto Sanyal added a comment -

        Attaching the patch for UI change

        Show
        Subroto Sanyal added a comment - Attaching the patch for UI change
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495345/MAPREDUCE-2990%281%29.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/809//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/809//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495345/MAPREDUCE-2990%281%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/809//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/809//console This message is automatically generated.
        Hide
        Subroto Sanyal added a comment -

        The changes impacts the UI, so no test included with the patch.

        Show
        Subroto Sanyal added a comment - The changes impacts the UI, so no test included with the patch.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        NodeManager shouldn't be passing null health. ever. This null-check done in the patch may be hiding some real bug somewhere. Can you please investigate? You can look in NodeStatusUpdaterImpl and ResourceTrackerService. Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - NodeManager shouldn't be passing null health. ever. This null-check done in the patch may be hiding some real bug somewhere. Can you please investigate? You can look in NodeStatusUpdaterImpl and ResourceTrackerService. Thanks!
        Hide
        Subroto Sanyal added a comment -

        1)
        The latest nodehealth is not updated at the RMNodeImpl at resourcemanager
        Only a event is triggered,in case of a node health status change.
        So no matter what's the status of the node, the node page in UI and the node report always shows the node as healthy as the status is never updated.
        2)
        Even if the NodeHealthChecker script exits with a non-zero exitcode the nodemanager reports the node as healthy.

        Show
        Subroto Sanyal added a comment - 1) The latest nodehealth is not updated at the RMNodeImpl at resourcemanager Only a event is triggered,in case of a node health status change. So no matter what's the status of the node, the node page in UI and the node report always shows the node as healthy as the status is never updated. 2) Even if the NodeHealthChecker script exits with a non-zero exitcode the nodemanager reports the node as healthy.
        Hide
        Subroto Sanyal added a comment -

        We have the above mentioned problems.

        Show
        Subroto Sanyal added a comment - We have the above mentioned problems.
        Hide
        Mahadev konar added a comment -

        @subroto,
        Are you working on fixing the above mentioned issues?

        Show
        Mahadev konar added a comment - @subroto, Are you working on fixing the above mentioned issues?
        Hide
        Subroto Sanyal added a comment -

        Hi Mahadev,

        I am working on this issue, will be providing a solution very soon....

        Show
        Subroto Sanyal added a comment - Hi Mahadev, I am working on this issue, will be providing a solution very soon....
        Hide
        Mahadev konar added a comment -

        thanks subroto!

        Show
        Mahadev konar added a comment - thanks subroto!
        Hide
        Subroto Sanyal added a comment -

        One more point to check here is:

        case FAILED_WITH_EXIT_CODE:
                setHealthStatus(true, "", now);
                break;
        

        The API has following documentation snippet:

             * If the script throws {@link IOException} or {@link ExitCodeException} the
             * output is ignored and node is left remaining healthy, as script might
             * have syntax error.
        

        If this is the case, then user may not come to about the problem in script.
        In my opinion the node health should be reported as unhealthy; which will attract the attention of the Script Developer/User

        Show
        Subroto Sanyal added a comment - One more point to check here is: case FAILED_WITH_EXIT_CODE: setHealthStatus( true , "", now); break ; The API has following documentation snippet: * If the script throws {@link IOException} or {@link ExitCodeException} the * output is ignored and node is left remaining healthy, as script might * have syntax error. If this is the case, then user may not come to about the problem in script. In my opinion the node health should be reported as unhealthy; which will attract the attention of the Script Developer/User
        Hide
        Vinod Kumar Vavilapalli added a comment -

        In my opinion the node health should be reported as unhealthy; which will attract the attention of the Script Developer/User

        This isn't how the feature was originally implemented, problems in the script don't imply a problem in the node. The node is deemed to be unhealthy only if the script successfully finishes and spits output in ERROR_PATTERN. This is how the feature is used in production code in MRV1, let us not change this.

        Show
        Vinod Kumar Vavilapalli added a comment - In my opinion the node health should be reported as unhealthy; which will attract the attention of the Script Developer/User This isn't how the feature was originally implemented, problems in the script don't imply a problem in the node. The node is deemed to be unhealthy only if the script successfully finishes and spits output in ERROR_PATTERN. This is how the feature is used in production code in MRV1, let us not change this.
        Hide
        Subroto Sanyal added a comment -

        Hi Vinod/Mahadev,

        In case if node is healthy, the node page table will look something like this:

        Rack Node Address Node HTTP Address Health-status Last health-update Health-report Containers
        /default-rack 10.18.52.110:45454 10.18.52.110:9999 Healthy 23-Sep-2011 14:12:58 Healthy

        Both the Columns Health-Status and Health-Report will display the node as Healthy in case is Node is running fine. Is it OK to have same text in both columns or we can have no text in second column if Health-Status is Healthy?

        Show
        Subroto Sanyal added a comment - Hi Vinod/Mahadev, In case if node is healthy, the node page table will look something like this: Rack Node Address Node HTTP Address Health-status Last health-update Health-report Containers /default-rack 10.18.52.110:45454 10.18.52.110:9999 Healthy 23-Sep-2011 14:12:58 Healthy Both the Columns Health-Status and Health-Report will display the node as Healthy in case is Node is running fine. Is it OK to have same text in both columns or we can have no text in second column if Health-Status is Healthy ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Is it OK to have same text in both columns or we can have no text in second column if Health-Status is Healthy?

        That depends on the implementation, some health-checker script can decide to set something else even when it is healthy.

        It should be fine to have both the columns with the same text in the default case.

        Show
        Vinod Kumar Vavilapalli added a comment - Is it OK to have same text in both columns or we can have no text in second column if Health-Status is Healthy? That depends on the implementation, some health-checker script can decide to set something else even when it is healthy. It should be fine to have both the columns with the same text in the default case.
        Hide
        Subroto Sanyal added a comment -

        Attached patch.
        This patch will make sure to set the NodeHealthStatus stored in RMNodeImpl with every hartbeat...

        Show
        Subroto Sanyal added a comment - Attached patch. This patch will make sure to set the NodeHealthStatus stored in RMNodeImpl with every hartbeat...
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12496273/MAPREDUCE-2990.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/838//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/838//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496273/MAPREDUCE-2990.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/838//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/838//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        +1 looks good.

        Minor changes - called it setNodeHealthStatus to be consistent with the getter and added the writeLock for clarity.

        Show
        Arun C Murthy added a comment - +1 looks good. Minor changes - called it setNodeHealthStatus to be consistent with the getter and added the writeLock for clarity.
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Subroto!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Subroto!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #950 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/950/)
        MAPREDUCE-2990. Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #950 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/950/ ) MAPREDUCE-2990 . Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1028/)
        MAPREDUCE-2990. Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1028/ ) MAPREDUCE-2990 . Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #968 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/968/)
        MAPREDUCE-2990. Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #968 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/968/ ) MAPREDUCE-2990 . Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #841 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/841/)
        MAPREDUCE-2990. Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #841 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/841/ ) MAPREDUCE-2990 . Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #811 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/811/)
        MAPREDUCE-2990. Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #811 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/811/ ) MAPREDUCE-2990 . Fixed display of NodeHealthStatus. Contributed by Subroto Sanyal. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175351 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java

          People

          • Assignee:
            Subroto Sanyal
            Reporter:
            Mahadev konar
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development