Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-492

Expose corrupt replica/block information

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New server web pages provide block information: corrupt_replicas_xml and block_info_xml.
    • Tags:
      block forensic blocks corrupt replica

      Description

      This adds two additional functions to FSNamesystem to provide more information about corrupt replicas. It also adds two servlets to the namenode that provide information (in JSON) about all blocks with corrupt replicas as well as information about a specific block. It also changes the file browsing servlet by adding a link from block ids to the above mentioned block information page.

      These JSON pages are designed to be used by client side tools which wish to analyze corrupt block/replicas. The only change to an existing (non-servlet) class is described below.

      Currently, CorruptReplicasMap stores a map of corrupt replica information and allows insertion and deletion. It also gives information about the corrupt replicas for a specific block. It does not allow iteration over all corrupt blocks. Two additional functions will be added to FSNamesystem (which will call BlockManager which will call CorruptReplicasMap). The first will return the size of the corrupt replicas map, which represents the number of blocks that have corrupt replicas (but less than the number of corrupt replicas if a block has multiple corrupt replicas). The second will allow "paging" through a list of block ids that contain corrupt replicas:

      public synchronized List<Long> getCorruptReplicaBlockIds(int n, Long startingBlockId)

      n is the number of block ids to return and startingBlockId is the block id offset. To prevent a large number of items being returned at one time, n is constrained to 0 <= n <= 100. If startingBlockId is null, up to n items are returned starting at the beginning of the list. Ordering is enforced through the internal use of TreeMap in CorruptReplicasMap.

      1. hdfs-492-4.patch
        22 kB
        Bill Zeller
      2. hdfs-492-5.patch
        22 kB
        Bill Zeller
      3. hdfs-492-8.patch
        22 kB
        Bill Zeller
      4. hdfs-492-9.patch
        25 kB
        Bill Zeller
      5. hdfs-492-10.patch
        25 kB
        Bill Zeller
      6. hdfs-492-11.patch
        25 kB
        Bill Zeller
      7. hdfs-492-13.patch
        26 kB
        Bill Zeller

        Activity

        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/)
        . Add two JSON JSP pages to the Namenode for providing corrupt blocks/replicas information. Contributed by Bill Zeller

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/ ) . Add two JSON JSP pages to the Namenode for providing corrupt blocks/replicas information. Contributed by Bill Zeller
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #5 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/5/)
        . Add two JSON JSP pages to the Namenode for providing corrupt blocks/replicas information. Contributed by Bill Zeller

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #5 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/5/ ) . Add two JSON JSP pages to the Namenode for providing corrupt blocks/replicas information. Contributed by Bill Zeller
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I forgot to say that the failed test, TestBackupNode, is not related to this. See HDFS-192.

        Show
        Tsz Wo Nicholas Sze added a comment - I forgot to say that the failed test, TestBackupNode, is not related to this. See HDFS-192 .
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Bill!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Bill!
        Hide
        Suresh Srinivas added a comment -

        Thanks Bill for addressing all the comments. +1 for the patch.

        Show
        Suresh Srinivas added a comment - Thanks Bill for addressing all the comments. +1 for the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417941/hdfs-492-13.patch
        against trunk revision 808670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417941/hdfs-492-13.patch against trunk revision 808670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417928/hdfs-492-11.patch
        against trunk revision 808193.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417928/hdfs-492-11.patch against trunk revision 808193. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/console This message is automatically generated.
        Hide
        Bill Zeller added a comment -

        I addressed Nicholas' issues.

        Show
        Bill Zeller added a comment - I addressed Nicholas' issues.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        In our Coding Style, "{" should be place in the same line. For examples,

        +  public long[] getCorruptReplicaBlockIds(int numExpectedBlocks,
        +                                              Long startingBlockId)
        +  {
        

        should be

        +  public long[] getCorruptReplicaBlockIds(int numExpectedBlocks,
        +                                              Long startingBlockId) {
        

        Also, could you remove the tab characters? We do not use tabs.

        Show
        Tsz Wo Nicholas Sze added a comment - In our Coding Style , "{" should be place in the same line. For examples, + public long [] getCorruptReplicaBlockIds( int numExpectedBlocks, + Long startingBlockId) + { should be + public long [] getCorruptReplicaBlockIds( int numExpectedBlocks, + Long startingBlockId) { Also, could you remove the tab characters? We do not use tabs.
        Hide
        Bill Zeller added a comment -

        The failed test is unrelated to this patch. See:
        https://issues.apache.org/jira/browse/HDFS-568

        Show
        Bill Zeller added a comment - The failed test is unrelated to this patch. See: https://issues.apache.org/jira/browse/HDFS-568
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417821/hdfs-492-9.patch
        against trunk revision 808193.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417821/hdfs-492-9.patch against trunk revision 808193. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch

        Show
        Suresh Srinivas added a comment - +1 for the patch
        Hide
        Bill Zeller added a comment -

        In regards to (1), doesn't toArray() require an object array? I don't believe this method supports primitive data types.

        Show
        Bill Zeller added a comment - In regards to (1), doesn't toArray() require an object array? I don't believe this method supports primitive data types.
        Hide
        Suresh Srinivas added a comment -
        1. In CorruptReplicasMap.getCorruptReplicaBlockIds() you can return array by code:
          return corruptReplicaBlockIds.toArray(new long[corruptReplicasBlockIds.size();
          
        2. Change methods TestCorruptReplicaInfo.getBlock() to private
        3. block_info_xml.jsp - XMLBlockInfo.getLocalPath() use StringBuilder instead of StringBuffer
        4. block_info_xml.jsp, corrupt_replicas_xml.jsp - could you please add comments to indicate what these jsps do
        5. XMLBlockInfo constructor - should fsn be set to the parameter to constructor irrespective of whether blockId argument is null or not?
        6. Rename XMLBlockInfo.getLocalPath() appropriately to indicate it is the parent directory that the method returns
        Show
        Suresh Srinivas added a comment - In CorruptReplicasMap.getCorruptReplicaBlockIds() you can return array by code: return corruptReplicaBlockIds.toArray(new long[corruptReplicasBlockIds.size(); Change methods TestCorruptReplicaInfo.getBlock() to private block_info_xml.jsp - XMLBlockInfo.getLocalPath() use StringBuilder instead of StringBuffer block_info_xml.jsp, corrupt_replicas_xml.jsp - could you please add comments to indicate what these jsps do XMLBlockInfo constructor - should fsn be set to the parameter to constructor irrespective of whether blockId argument is null or not? Rename XMLBlockInfo.getLocalPath() appropriately to indicate it is the parent directory that the method returns
        Hide
        Bill Zeller added a comment -

        Ran Hudson tests locally:

        [exec] There appear to be 147 release audit warnings before the patch and 147 release audit warnings after applying the patch.
        [exec]
        [exec] +1 overall.
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec] +1 tests included. The patch appears to include 2 new or modified tests.
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] ======================================================================
        [exec] ======================================================================
        [exec] Finished build.
        [exec] ======================================================================
        [exec] ======================================================================
        [exec]

        BUILD SUCCESSFUL

        (some blank lines removed)

        Running ant run-test-hdfs:

        BUILD SUCCESSFUL
        Total time: 36 minutes 1 second

        Show
        Bill Zeller added a comment - Ran Hudson tests locally: [exec] There appear to be 147 release audit warnings before the patch and 147 release audit warnings after applying the patch. [exec] [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] BUILD SUCCESSFUL (some blank lines removed) Running ant run-test-hdfs: BUILD SUCCESSFUL Total time: 36 minutes 1 second
        Hide
        Bill Zeller added a comment -

        The second argument is the startingBlockId, not the starting block offset. Passing 0 indicates a block id of 0. This function could be special cased to treat 0 to mean to start from the beginning and assume that no real block id would ever have the value 0, but this seems to be dirtier than just using null.

        Show
        Bill Zeller added a comment - The second argument is the startingBlockId, not the starting block offset. Passing 0 indicates a block id of 0. This function could be special cased to treat 0 to mean to start from the beginning and assume that no real block id would ever have the value 0, but this seems to be dirtier than just using null .
        Hide
        Suresh Srinivas added a comment -

        long passed to the method can be set to 0 right to start from the beginning?

        Show
        Suresh Srinivas added a comment - long passed to the method can be set to 0 right to start from the beginning?
        Hide
        Bill Zeller added a comment -

        I made all the above changes except for (4).

        The reason I take a Long as the startingBlockId parameter (instead of a long), is to allow null to be passed. If null is passed as startingBlockId, the iteration begins at the beginning of the corrupt replica list. If null is not passed, the iteration begins at the block following startingBlockId. I couldn't think of a clean way to do this other than to create another method which duplicates some of the functionality. Another option would be to create a third method which takes an iterator and n and returns up to n blocks wherever that iterator begins. Two methods could be created calling this third method (one which advanced the iterator to the right block and the other which passed an iterator which hadn't been advanced at all). Adding another method would require propagating it to BlockManager and FSNameSystem, which I thought might be too heavyweight.

        Show
        Bill Zeller added a comment - I made all the above changes except for (4). The reason I take a Long as the startingBlockId parameter (instead of a long), is to allow null to be passed. If null is passed as startingBlockId, the iteration begins at the beginning of the corrupt replica list. If null is not passed, the iteration begins at the block following startingBlockId. I couldn't think of a clean way to do this other than to create another method which duplicates some of the functionality. Another option would be to create a third method which takes an iterator and n and returns up to n blocks wherever that iterator begins. Two methods could be created calling this third method (one which advanced the iterator to the right block and the other which passed an iterator which hadn't been advanced at all). Adding another method would require propagating it to BlockManager and FSNameSystem, which I thought might be too heavyweight.
        Hide
        Suresh Srinivas added a comment -

        Comments on the previous version of the patch. I will review the next version that works with trunk:

        1. BlockManager.numBlocksWithCorruptReplicas()
          • Currently this can be obtained by by looking BlockManager.corruptReplicaBlocksCount (unsynchronized). However this count is updated every dfs.replication.interva. If that approximate number is sufficent, then there is no need to introduce this method. Also existing method FSNamesystem.getCorruptReplicaBlocks() instead of new method added FSNamesystem.numBlocksWithCorruptReplicas(), can be used.
          • If the number needs to be accurate, then remove BlockManager.corruptReplicaBlocksCount and synchronize the method FSNamesystem.getCorruptReplicaBlocks() and use BlockManager.numBlocksWithCorruptReplicas() to get the count.
        2. CorruptReplicasMap.numBlocksWithCorruptReplicas() - instead of adding this, use existing method CorruptReplicasMap.size()
        3. Comments in CorruptReplicasMap.java can be shorted. No need to explain how the code works - a brief comment will be sufficient . Also remove the comment with XXXBZ
        4. CorruptReplicasMap.getCorruptReplicaBlockIds - it is better to take long as second param instead of object Long. Also returning long[] is better than List<Long>
        5. TestCorruptReplicaInfo.java - use assertEquals() instead of assertTrue(). When the test fails it gives information about expected Vs actual value.
        6. TestCorruptReplicaInfo.block_map - can just be a HashMap

        Nits:

        1. Generic - instead of n, use numExpectedBlocks
        2. CorruptReplicasMap.java - better to change the type of corruptReplicasMap from TreeMap to SortedMap
        Show
        Suresh Srinivas added a comment - Comments on the previous version of the patch. I will review the next version that works with trunk: BlockManager.numBlocksWithCorruptReplicas() Currently this can be obtained by by looking BlockManager.corruptReplicaBlocksCount (unsynchronized). However this count is updated every dfs.replication.interva . If that approximate number is sufficent, then there is no need to introduce this method. Also existing method FSNamesystem.getCorruptReplicaBlocks() instead of new method added FSNamesystem.numBlocksWithCorruptReplicas() , can be used. If the number needs to be accurate, then remove BlockManager.corruptReplicaBlocksCount and synchronize the method FSNamesystem.getCorruptReplicaBlocks() and use BlockManager.numBlocksWithCorruptReplicas() to get the count. CorruptReplicasMap.numBlocksWithCorruptReplicas() - instead of adding this, use existing method CorruptReplicasMap.size() Comments in CorruptReplicasMap.java can be shorted. No need to explain how the code works - a brief comment will be sufficient . Also remove the comment with XXXBZ CorruptReplicasMap.getCorruptReplicaBlockIds - it is better to take long as second param instead of object Long . Also returning long[] is better than List<Long> TestCorruptReplicaInfo.java - use assertEquals() instead of assertTrue(). When the test fails it gives information about expected Vs actual value. TestCorruptReplicaInfo.block_map - can just be a HashMap Nits: Generic - instead of n , use numExpectedBlocks CorruptReplicasMap.java - better to change the type of corruptReplicasMap from TreeMap to SortedMap
        Hide
        Bill Zeller added a comment -

        The change to BlockInfo broke this patch. I need to make it work with the current code.

        Show
        Bill Zeller added a comment - The change to BlockInfo broke this patch. I need to make it work with the current code.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12414283/hdfs-492-5.patch
        against trunk revision 799146.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/testReport/
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414283/hdfs-492-5.patch against trunk revision 799146. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12413885/hdfs-492-4.patch
        against trunk revision 794953.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs warnings.

        -1 release audit. The applied patch generated 283 release audit warnings (more than the trunk's current 282 warnings).

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/testReport/
        Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/current/releaseAuditDiffWarnings.txt
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413885/hdfs-492-4.patch against trunk revision 794953. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs warnings. -1 release audit. The applied patch generated 283 release audit warnings (more than the trunk's current 282 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/current/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        This utility will be be a great help to monitor and process corrupt blocks in the file system.

        Is it possible to also make HDFS throw a CorruptBlockException (which can be a subclass of IOException) when an app tries to read a block that is corrupt. This helps in developing intelligent clients that can do a variety of things for this type of problem: maybe retrieve the file from an external location (maybe from tape) or from an alternate datacenter (if it is replicated there).

        Show
        dhruba borthakur added a comment - This utility will be be a great help to monitor and process corrupt blocks in the file system. Is it possible to also make HDFS throw a CorruptBlockException (which can be a subclass of IOException) when an app tries to read a block that is corrupt. This helps in developing intelligent clients that can do a variety of things for this type of problem: maybe retrieve the file from an external location (maybe from tape) or from an alternate datacenter (if it is replicated there).

          People

          • Assignee:
            Bill Zeller
            Reporter:
            Bill Zeller
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development