Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1533

A more elegant FileSystem#listCorruptFileBlocks API (HDFS portion)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is the HDFS portion of HADOOP-7060.

      1. HDFS-1533.patch
        12 kB
        Patrick Kling
      2. HDFS-1533.2.patch
        21 kB
        Patrick Kling

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-2215 [ MAPREDUCE-2215 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-2215 [ MAPREDUCE-2215 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-7060 [ HADOOP-7060 ]
          Gavin made changes -
          Link This issue depends on HADOOP-7060 [ HADOOP-7060 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hairong Kuang made changes -
          Affects Version/s 0.23.0 [ 12315571 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315571 ]
          Resolution Fixed [ 1 ]
          Hide
          Hairong Kuang added a comment -

          I've committed this. Thanks, Patrick!

          Show
          Hairong Kuang added a comment - I've committed this. Thanks, Patrick!
          Hide
          Konstantin Boudnik added a comment -

          Shall we commit this because trunk is broken after HADOOP-7060 was committed

          Show
          Konstantin Boudnik added a comment - Shall we commit this because trunk is broken after HADOOP-7060 was committed
          Hide
          Hairong Kuang added a comment -

          +1. This looks good to me.

          Show
          Hairong Kuang added a comment - +1. This looks good to me.
          Patrick Kling made changes -
          Attachment HDFS-1533.2.patch [ 12465943 ]
          Hide
          Patrick Kling added a comment -

          Moved CorruptFileBlocks and TestCorruptFileBlocks to HDFS.

          Verified that ant test/ant test-patch results did not change.

          Show
          Patrick Kling added a comment - Moved CorruptFileBlocks and TestCorruptFileBlocks to HDFS. Verified that ant test/ant test-patch results did not change.
          Hide
          Hairong Kuang added a comment -

          Oh, I just realized that the patch that populating the replication queue earlier has not been committed yet.

          Show
          Hairong Kuang added a comment - Oh, I just realized that the patch that populating the replication queue earlier has not been committed yet.
          Hide
          Hairong Kuang added a comment -

          It looks good except that
          1. Move class CorruptFileBlocks to package hdfs.protocol;
          2. When handling listCorruptFileBlocks in safemode, the exception thrown condition should be the replication queue not populated yet.

          Show
          Hairong Kuang added a comment - It looks good except that 1. Move class CorruptFileBlocks to package hdfs.protocol; 2. When handling listCorruptFileBlocks in safemode, the exception thrown condition should be the replication queue not populated yet.
          Patrick Kling made changes -
          Summary A more elegant FileSystem#listCorruptFileBlocks AP (HDFS portion) A more elegant FileSystem#listCorruptFileBlocks API (HDFS portion)
          Patrick Kling made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Kling made changes -
          Attachment HDFS-1533.patch [ 12465883 ]
          Hide
          Patrick Kling added a comment -

          ant test yields the following errors (same as on clean trunk):

              [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
              [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED
              [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestStorageRestore FAILED
              [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancer FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
          

          ant test-patch output:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Patrick Kling added a comment - ant test yields the following errors (same as on clean trunk): [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestStorageRestore FAILED [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancer FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED ant test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.
          Patrick Kling made changes -
          Link This issue blocks MAPREDUCE-2215 [ MAPREDUCE-2215 ]
          Patrick Kling made changes -
          Description This is the HDFS portion of HADOOP-7060.
          Patrick Kling made changes -
          Field Original Value New Value
          Link This issue depends on HADOOP-7060 [ HADOOP-7060 ]
          Patrick Kling created issue -

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development