Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-567

Two contrib tools to facilitate searching for block history information

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New contribution Block Forensics aids investigation of missing blocks.

      Description

      Includes a java program to query the namenode for corrupt replica information at some interval. If a corrupt replica is found, a map reduce job is launched that will search (supplied) log files for one or more block ids. The mapred job can be used independently of the java client program and can also be used for arbitrary text searches.

      1. hdfs-567-6.patch
        19 kB
        Jitendra Nath Pandey
      2. hdfs-567-5.patch
        19 kB
        Bill Zeller
      3. hdfs-567-4.patch
        17 kB
        Bill Zeller
      4. hdfs-567-3.patch
        17 kB
        Bill Zeller
      5. hdfs-567-2.patch
        17 kB
        Bill Zeller
      6. hdfs-567-1.patch
        16 kB
        Bill Zeller

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417643/hdfs-567-1.patch
        against trunk revision 807433.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        -1 release audit. The applied patch generated 149 release audit warnings (more than the trunk's current 147 warnings).

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/testReport/
        Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417643/hdfs-567-1.patch against trunk revision 807433. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 149 release audit warnings (more than the trunk's current 147 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/86/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417691/hdfs-567-2.patch
        against trunk revision 807818.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417691/hdfs-567-2.patch against trunk revision 807818. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/89/console This message is automatically generated.
        Hide
        Bill Zeller added a comment -

        The failed test is unrelated to this patch. See:
        https://issues.apache.org/jira/browse/HDFS-568

        Show
        Bill Zeller added a comment - The failed test is unrelated to this patch. See: https://issues.apache.org/jira/browse/HDFS-568
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418197/hdfs-567-4.patch
        against trunk revision 809439.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418197/hdfs-567-4.patch against trunk revision 809439. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/4/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -
        1. README: Add an example for the mapreduce utility and BlockForensics utility in the README
        2. README: should sleep time between requests be sleep time between namenode query for corrupt blocks.
        3. BlockForensics.RunHadoopCmd() - is there a significance in starting the method name with caps?
        4. BlockForensics.RunHadoopCmd() - in case hadoop_home is not defined, should the command fail by printing System.err message? Also it might be good idea to check this early in main() itself.
        5. BlockForensics.main() - when sleep time parse fails, a message is printed indicating the default value is being used. Can you please print the default value in that message?
        6. BlocksLog and BlockForensics - could you please add comments to the class and brief description of the functionality
        Show
        Suresh Srinivas added a comment - README: Add an example for the mapreduce utility and BlockForensics utility in the README README: should sleep time between requests be sleep time between namenode query for corrupt blocks . BlockForensics.RunHadoopCmd() - is there a significance in starting the method name with caps? BlockForensics.RunHadoopCmd() - in case hadoop_home is not defined, should the command fail by printing System.err message? Also it might be good idea to check this early in main() itself. BlockForensics.main() - when sleep time parse fails, a message is printed indicating the default value is being used. Can you please print the default value in that message? BlocksLog and BlockForensics - could you please add comments to the class and brief description of the functionality
        Hide
        Bill Zeller added a comment -

        I addressed Suresh's six issues above.

        Show
        Bill Zeller added a comment - I addressed Suresh's six issues above.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418550/hdfs-567-5.patch
        against trunk revision 810631.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418550/hdfs-567-5.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/10/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1

        Show
        Suresh Srinivas added a comment - +1
        Hide
        Suresh Srinivas added a comment -

        I noticed that the contrib is called corrupted_blocks. The name does not seem appropriate. It needs to be changed to block_forensics. Also package o.a.h.corrupted_blocks needs to be changed to o.a.h.blockforensics. The class BlocksLog could be renamed as BlockSearch.

        Show
        Suresh Srinivas added a comment - I noticed that the contrib is called corrupted_blocks. The name does not seem appropriate. It needs to be changed to block_forensics. Also package o.a.h.corrupted_blocks needs to be changed to o.a.h.blockforensics. The class BlocksLog could be renamed as BlockSearch.
        Hide
        Suresh Srinivas added a comment -

        Commit this contrib tool. Thanks Bill and Jithendra.

        Show
        Suresh Srinivas added a comment - Commit this contrib tool. Thanks Bill and Jithendra.
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #35 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/35/)
        . Add block forensics contrib tool to print history of corrupt and missing blocks from the HDFS logs. Contributed by Bill Zeller and Jithendra Pandey.

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #35 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/35/ ) . Add block forensics contrib tool to print history of corrupt and missing blocks from the HDFS logs. Contributed by Bill Zeller and Jithendra Pandey.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #44 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/44/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #44 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/44/ )
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #98 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/98/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #98 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/98/ )
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Bill Zeller
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 5h
              5h
              Remaining:
              Remaining Estimate - 5h
              5h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development