Hadoop Common
  1. Hadoop Common
  2. HADOOP-6216

HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I am putting comments in dfs.exclude file such as:

       # 32 GB memory upgrades
      

      HDFS Web UI counts each word in the commented line as a dead node.

      1. HDFS-1.patch
        0.9 kB
        Ravi Phulari
      2. HADOOP-6216.patch
        4 kB
        Ravi Phulari
      3. HADOOP-6216.patch
        7 kB
        Ravi Phulari
      4. HADOOP-6216.patch
        8 kB
        Dmytro Molkov
      5. HADOOP-6216.patch
        8 kB
        Dmytro Molkov

        Issue Links

          Activity

          Hide
          Lohit Vijayarenu added a comment -

          This happens because hostsReader::excludes is added to the report in FSNamesystem::getDatanodeListForReport

           for (Iterator<String> it = hostsReader.getExcludedHosts().iterator(); 
                     it.hasNext();) {
                  mustList.put(it.next(), "");
                }
          

          When dfs.exclude.file is parsed, they are just added to internal array list without validating.
          We should cleanup the list in refreshNodes().

          Show
          Lohit Vijayarenu added a comment - This happens because hostsReader::excludes is added to the report in FSNamesystem::getDatanodeListForReport for (Iterator<String> it = hostsReader.getExcludedHosts().iterator(); it.hasNext();) { mustList.put(it.next(), ""); } When dfs.exclude.file is parsed, they are just added to internal array list without validating. We should cleanup the list in refreshNodes().
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, the codes actually belong to Common although the problem was observed in HDFS. Will move the issue back to Common.

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, the codes actually belong to Common although the problem was observed in HDFS. Will move the issue back to Common.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          How about changing line.startsWith("#") to line.trim().startsWith("#")? Also, could you use our Coding Style?

          Show
          Tsz Wo Nicholas Sze added a comment - How about changing line.startsWith("#") to line.trim().startsWith("#")? Also, could you use our Coding Style ?
          Hide
          Ravi Phulari added a comment -

          Nicolas,Thanks for reviewing.
          Attaching updated patch with unit test and changes suggested by Nicolas.

          Show
          Ravi Phulari added a comment - Nicolas,Thanks for reviewing. Attaching updated patch with unit test and changes suggested by Nicolas.
          Hide
          Dmytro Molkov added a comment -

          What about adding inline comments handling?
          You could just perform the startsWith("#") in the for loop iterating through the String[] nodes, breaking on the # to get the next line.

          Show
          Dmytro Molkov added a comment - What about adding inline comments handling? You could just perform the startsWith("#") in the for loop iterating through the String[] nodes, breaking on the # to get the next line.
          Hide
          Dmytro Molkov added a comment -

          I added the possibility of inline comments (hostname # reason for adding) and modified the testcase to reflect the changes.

          Show
          Dmytro Molkov added a comment - I added the possibility of inline comments (hostname # reason for adding) and modified the testcase to reflect the changes.
          Hide
          Dmytro Molkov added a comment -

          There was something terribly wrong with the tabs formatting in my previous attachment.

          Show
          Dmytro Molkov added a comment - There was something terribly wrong with the tabs formatting in my previous attachment.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419337/HADOOP-6216.patch
          against trunk revision 813698.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419337/HADOOP-6216.patch against trunk revision 813698. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/31/console This message is automatically generated.
          Hide
          Ravi Phulari added a comment -

          +1
          Thanks for updating patch Dmytro. Patch looks good.

          Show
          Ravi Phulari added a comment - +1 Thanks for updating patch Dmytro. Patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Ravi and Dmytro!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Ravi and Dmytro!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #33 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/33/)
          . Support comments in host files. Contributed by Ravi Phulari and Dmytro Molkov

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #33 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/33/ ) . Support comments in host files. Contributed by Ravi Phulari and Dmytro Molkov
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #97 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/97/)
          . Support comments in host files. Contributed by Ravi Phulari and Dmytro Molkov

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #97 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/97/ ) . Support comments in host files. Contributed by Ravi Phulari and Dmytro Molkov

            People

            • Assignee:
              Ravi Phulari
              Reporter:
              Jim Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development