Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3735

NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 2.0.1-alpha
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Live datanode list is not correctly sorted for columns Block Pool Used (GB),
      Block Pool Used (%) and Failed Volumes. Read comments for more details.

      1. HDFS-3735.patch
        5 kB
        Brahma Reddy Battula
      2. HDFS-3735-1.patch
        5 kB
        Brahma Reddy Battula

        Activity

        Hide
        sureshms Suresh Srinivas added a comment -

        Fields which will display on dfsnodelist.jsp?whatNodes=LIVE ( NNUI->click on live nodes)

        Node| Last-contact |Admin State| Configured Capacity(GB)|Used(GB)|

        Non DFSUsed(GB)Remaining(GB)|Used(%)|Used(%)|Remaining(%)|Blocks|

        Block Pool Used (GB)|Block Pool Used (%)> Blocks|Failed Volumes H --> these three fields not considered for sorting

        we can display data in sorting order by clicking on the anyone of above fields.It's working fine for all fields expect last three fields.

        code where we are considering fileds.

        class NodeComapare implements Comparator<DatanodeDescriptor> {
              static final int 
                FIELD_NAME              = 1,
                FIELD_LAST_CONTACT      = 2,
                FIELD_BLOCKS            = 3,
                FIELD_CAPACITY          = 4,
                FIELD_USED              = 5,
                FIELD_PERCENT_USED      = 6,
                FIELD_NONDFS_USED       = 7,
                FIELD_REMAINING         = 8,
                FIELD_PERCENT_REMAINING = 9,
                FIELD_ADMIN_STATE       = 10,
                FIELD_DECOMMISSIONED    = 11,
                SORT_ORDER_ASC          = 1,
                SORT_ORDER_DSC          = 2;
        

        Here,last three fields we are not considering..hence it's assign default filed.

         } else if (field.equals("blocks")) {
                  sortField = FIELD_BLOCKS;
                } else if (field.equals("adminstate")) {
                  sortField = FIELD_ADMIN_STATE;
                } else if (field.equals("decommissioned")) {
                  sortField = FIELD_DECOMMISSIONED;
                } else {
                  sortField = FIELD_NAME;
                }
        

        Please correct me ,If I am wrong...

        Show
        sureshms Suresh Srinivas added a comment - Fields which will display on dfsnodelist.jsp?whatNodes=LIVE ( NNUI->click on live nodes) Node| Last-contact |Admin State| Configured Capacity(GB)|Used(GB)| Non DFSUsed(GB)Remaining(GB)|Used(%)|Used(%)|Remaining(%)|Blocks| Block Pool Used (GB)|Block Pool Used (%)> Blocks|Failed Volumes H --> these three fields not considered for sorting we can display data in sorting order by clicking on the anyone of above fields.It's working fine for all fields expect last three fields. code where we are considering fileds. class NodeComapare implements Comparator<DatanodeDescriptor> { static final int FIELD_NAME = 1, FIELD_LAST_CONTACT = 2, FIELD_BLOCKS = 3, FIELD_CAPACITY = 4, FIELD_USED = 5, FIELD_PERCENT_USED = 6, FIELD_NONDFS_USED = 7, FIELD_REMAINING = 8, FIELD_PERCENT_REMAINING = 9, FIELD_ADMIN_STATE = 10, FIELD_DECOMMISSIONED = 11, SORT_ORDER_ASC = 1, SORT_ORDER_DSC = 2; Here,last three fields we are not considering..hence it's assign default filed. } else if (field.equals( "blocks" )) { sortField = FIELD_BLOCKS; } else if (field.equals( "adminstate" )) { sortField = FIELD_ADMIN_STATE; } else if (field.equals( "decommissioned" )) { sortField = FIELD_DECOMMISSIONED; } else { sortField = FIELD_NAME; } Please correct me ,If I am wrong...
        Hide
        sureshms Suresh Srinivas added a comment -

        Brahma, can you please keep the description of the jira short and add the details in a comment later.

        Your analysis seems to be correct. Please post a patch, I will review it and commit it.

        Show
        sureshms Suresh Srinivas added a comment - Brahma, can you please keep the description of the jira short and add the details in a comment later. Your analysis seems to be correct. Please post a patch, I will review it and commit it.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Hi Suresh,

        Thanks for taking look,I have fixed and verified manually.It's working fine.

        Added following in JspHelper.java

        method : NodeComapare

                } else if (field.equals("bpused")) {
                  sortField = FIELD_BP_USED;
                }else if (field.equals("pcbpused")) {
                  sortField = FIELD_PERCENT_BP_USED;
                }else if (field.equals("volfails")) {
                  sortField = FIELD_FAILED_VOL;
                }
        

        method : compare

         case FIELD_BP_USED:
                  dlong = d1.getBlockPoolUsed() - d2.getBlockPoolUsed();
                  ret = (dlong < 0) ? -1 : ((dlong > 0) ? 1 : 0);
                  break;
                case FIELD_PERCENT_BP_USED:
                   ddbl =((d1.getBlockPoolUsedPercent())-
                                (d2.getBlockPoolUsedPercent()));
                  ret = (ddbl < 0) ? -1 : ((ddbl > 0) ? 1 : 0);
                  break;
                case FIELD_FAILED_VOL:
                  ret = d1.getVolumeFailures() - d2.getVolumeFailures();
                  break;
        

        Will post the patch once testcase written(need to write).

        Show
        brahmareddy Brahma Reddy Battula added a comment - Hi Suresh, Thanks for taking look,I have fixed and verified manually.It's working fine. Added following in JspHelper.java method : NodeComapare } else if (field.equals( "bpused" )) { sortField = FIELD_BP_USED; } else if (field.equals( "pcbpused" )) { sortField = FIELD_PERCENT_BP_USED; } else if (field.equals( "volfails" )) { sortField = FIELD_FAILED_VOL; } method : compare case FIELD_BP_USED: dlong = d1.getBlockPoolUsed() - d2.getBlockPoolUsed(); ret = (dlong < 0) ? -1 : ((dlong > 0) ? 1 : 0); break ; case FIELD_PERCENT_BP_USED: ddbl =((d1.getBlockPoolUsedPercent())- (d2.getBlockPoolUsedPercent())); ret = (ddbl < 0) ? -1 : ((ddbl > 0) ? 1 : 0); break ; case FIELD_FAILED_VOL: ret = d1.getVolumeFailures() - d2.getVolumeFailures(); break ; Will post the patch once testcase written(need to write).
        Hide
        rajuvishnu Raju added a comment -

        Hey Brahma,

        case FIELD_FAILED_VOL:
                  ret = d1.getVolumeFailures() - d2.getVolumeFailures();
                  break;
        

        Since we are using this code in comparator its good to return -1,0 or 1 as its return value. Even without that with current fix it will work. But in general the mentioned values are returned.

        Show
        rajuvishnu Raju added a comment - Hey Brahma, case FIELD_FAILED_VOL: ret = d1.getVolumeFailures() - d2.getVolumeFailures(); break ; Since we are using this code in comparator its good to return -1,0 or 1 as its return value. Even without that with current fix it will work. But in general the mentioned values are returned.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Hey Suresh And Raju,

        Please review the patch...I really forgot this issue..sorry for late response..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Hey Suresh And Raju, Please review the patch...I really forgot this issue..sorry for late response..
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12546285/HDFS-3735.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3242//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546285/HDFS-3735.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3242//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12547669/HDFS-3735-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3264//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3264//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547669/HDFS-3735-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3264//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3264//console This message is automatically generated.
        Hide
        sureshms Suresh Srinivas added a comment -

        +1 for the patch.

        Show
        sureshms Suresh Srinivas added a comment - +1 for the patch.
        Hide
        sureshms Suresh Srinivas added a comment -

        Brahma Reddy, I have added you as a contributor to HDFS and Common. You can now assign bugs to yourself.

        Show
        sureshms Suresh Srinivas added a comment - Brahma Reddy, I have added you as a contributor to HDFS and Common. You can now assign bugs to yourself.
        Hide
        sureshms Suresh Srinivas added a comment -

        I committed the patch to trunk. Thank you Brahma for the jira and the patch.

        Show
        sureshms Suresh Srinivas added a comment - I committed the patch to trunk. Thank you Brahma for the jira and the patch.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2878 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2878/)
        HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2878 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2878/ ) HDFS-3735 . NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2816 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2816/)
        HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2816 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2816/ ) HDFS-3735 . NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2840/)
        HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385)

        Result = FAILURE
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2840/ ) HDFS-3735 . NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1186/)
        HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1186/ ) HDFS-3735 . NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1217 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1217/)
        HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1217 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1217/ ) HDFS-3735 . NameNode WebUI should allow sorting live datanode list by fields Block Pool Used, Block Pool Used(%) and Failed Volumes. Contributed by Brahma Reddy Battula. (Revision 1394385) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1394385 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            brahmareddy Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development