Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5748

Too much information shown in the dfs health page.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've noticed that the node lists are shown in the default name node web page. This may be fine for small clusters, but for clusters with 1000s of nodes, this is not ideal. The following should be shown on demand. (Some of them have been there even before the recent rework.)

      • Detailed data node information
      • Startup progress
      • Snapshot information
      1. hdfs-5748.png
        236 kB
        Haohui Mai
      2. HDFS-5748.000.patch
        24 kB
        Haohui Mai
      3. HDFS-5748.001.patch
        24 kB
        Haohui Mai

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1652 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1652/)
          HDFS-5748. Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1652 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1652/ ) HDFS-5748 . Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1677 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1677/)
          HDFS-5748. Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1677 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1677/ ) HDFS-5748 . Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #460 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/460/)
          HDFS-5748. Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #460 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/460/ ) HDFS-5748 . Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5030/)
          HDFS-5748. Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5030/ ) HDFS-5748 . Too much information shown in the dfs health page. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560215 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
          Hide
          Brandon Li added a comment -

          I've committed the patch. Thank you, Haohui and Kihwal.

          Show
          Brandon Li added a comment - I've committed the patch. Thank you, Haohui and Kihwal.
          Hide
          Brandon Li added a comment -

          +1.

          Show
          Brandon Li added a comment - +1.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623716/HDFS-5748.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5915//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5915//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623716/HDFS-5748.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5915//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5915//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          When viewing the datanodes tab, jmx query of "Hadoop:service=NameNode,name=NameNodeInfo" is made three times. This was already retrieved when dfshealth.html was initially loaded.

          I cannot reproduce the behavior. It is intentional to load the JMX multiple times to keep things simpler.

          A refresh button might help.

          The new patch recognizes the URL fragment so you can refresh manually.

          Version info was dropped in the datanodes view. This is important for rolling upgrades.

          Fixed.

          Live and dead nodes need to be grouped together.

          I believe that they are grouped. Can you elaborate?

          Somehow the datanodes table is not populated with this patch.

          Fixed.

          Show
          Haohui Mai added a comment - When viewing the datanodes tab, jmx query of "Hadoop:service=NameNode,name=NameNodeInfo" is made three times. This was already retrieved when dfshealth.html was initially loaded. I cannot reproduce the behavior. It is intentional to load the JMX multiple times to keep things simpler. A refresh button might help. The new patch recognizes the URL fragment so you can refresh manually. Version info was dropped in the datanodes view. This is important for rolling upgrades. Fixed. Live and dead nodes need to be grouped together. I believe that they are grouped. Can you elaborate? Somehow the datanodes table is not populated with this patch. Fixed.
          Hide
          Kihwal Lee added a comment -

          I will point out few things I've noticed with firefox.

          • When viewing the datanodes tab, jmx query of "Hadoop:service=NameNode,name=NameNodeInfo" is made three times. This was already retrieved when dfshealth.html was initially loaded.
          • When viewing explorer.html, the page header (tabs) are not shown. The log page is not under NN's control, so it is okay this way for now. This page has other problems, but we can deal with them in a separate jira.
          • In the old UI, users could load up the datanode page and refresh manually. A refresh button might help.
          • Version info was dropped in the datanodes view. This is important for rolling upgrades.
          • Live and dead nodes need to be grouped together.
          • Lost ability to sort by each column. Adding this feature may solve the grouping problem.
          • Somehow the datanodes table is not populated with this patch.
          Show
          Kihwal Lee added a comment - I will point out few things I've noticed with firefox. When viewing the datanodes tab, jmx query of "Hadoop:service=NameNode,name=NameNodeInfo" is made three times. This was already retrieved when dfshealth.html was initially loaded. When viewing explorer.html, the page header (tabs) are not shown. The log page is not under NN's control, so it is okay this way for now. This page has other problems, but we can deal with them in a separate jira. In the old UI, users could load up the datanode page and refresh manually. A refresh button might help. Version info was dropped in the datanodes view. This is important for rolling upgrades. Live and dead nodes need to be grouped together. Lost ability to sort by each column. Adding this feature may solve the grouping problem. Somehow the datanodes table is not populated with this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623588/HDFS-5748.000.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5908//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5908//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623588/HDFS-5748.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5908//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5908//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          Convert information into tabs.

          Show
          Haohui Mai added a comment - Convert information into tabs.

            People

            • Assignee:
              Haohui Mai
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development