Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).

      1. MAPREDUCE-2156.patch
        41 kB
        Patrick Kling
      2. MAPREDUCE-2156.3.patch
        41 kB
        Patrick Kling
      3. MAPREDUCE-2156.2.patch
        40 kB
        Patrick Kling

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          35d 5h 17m 1 Patrick Kling 30/Nov/10 22:33
          Patch Available Patch Available Resolved Resolved
          2d 7h 26m 1 dhruba borthakur 03/Dec/10 06:00
          Resolved Resolved Closed Closed
          346d 18h 49m 1 Arun C Murthy 15/Nov/11 00:50
          Gavin made changes -
          Link This issue depends upon HDFS-1483 [ HDFS-1483 ]
          Gavin made changes -
          Link This issue depends on HDFS-1483 [ HDFS-1483 ]
          Gavin made changes -
          Link This issue depends upon MAPREDUCE-2111 [ MAPREDUCE-2111 ]
          Gavin made changes -
          Link This issue depends on MAPREDUCE-2111 [ MAPREDUCE-2111 ]
          Gavin made changes -
          Link This issue depends upon MAPREDUCE-2110 [ MAPREDUCE-2110 ]
          Gavin made changes -
          Link This issue depends on MAPREDUCE-2110 [ MAPREDUCE-2110 ]
          Gavin made changes -
          Link This issue depends upon MAPREDUCE-1752 [ MAPREDUCE-1752 ]
          Gavin made changes -
          Link This issue depends on MAPREDUCE-1752 [ MAPREDUCE-1752 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/)
          MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/ ) MAPREDUCE-2156 . Raid-aware FSCK. (Patrick Kling via dhruba)
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Patrick!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Patrick!
          Patrick Kling made changes -
          Attachment MAPREDUCE-2156.3.patch [ 12465157 ]
          Hide
          Patrick Kling added a comment -

          Updated README.

          Show
          Patrick Kling added a comment - Updated README.
          Hide
          Ramkumar Vadali added a comment -

          +1, looks good.

          Show
          Ramkumar Vadali added a comment - +1, looks good.
          Patrick Kling made changes -
          Attachment MAPREDUCE-2156.2.patch [ 12465002 ]
          Hide
          Patrick Kling added a comment -

          Removed commented-out code.

          ParityBlockInfo is no longer needed since we now use HarFileSystem.getFileBlockLocations() from MAPREDUCE-1752.

          Show
          Patrick Kling added a comment - Removed commented-out code. ParityBlockInfo is no longer needed since we now use HarFileSystem.getFileBlockLocations() from MAPREDUCE-1752 .
          Patrick Kling made changes -
          Component/s contrib/raid [ 12313416 ]
          Patrick Kling made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Assignee Patrick Kling [ pkling ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Patrick Kling made changes -
          Attachment MAPREDUCE-2156.patch [ 12464999 ]
          Hide
          Patrick Kling added a comment -

          See review board: https://reviews.apache.org/r/132/

          ant test-patch output:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Patrick Kling added a comment - See review board: https://reviews.apache.org/r/132/ ant test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.
          Patrick Kling made changes -
          Link This issue depends on HDFS-1483 [ HDFS-1483 ]
          Patrick Kling made changes -
          Link This issue depends on MAPREDUCE-1752 [ MAPREDUCE-1752 ]
          Patrick Kling made changes -
          Link This issue depends on MAPREDUCE-2111 [ MAPREDUCE-2111 ]
          Patrick Kling made changes -
          Field Original Value New Value
          Link This issue depends on MAPREDUCE-2110 [ MAPREDUCE-2110 ]
          Patrick Kling created issue -

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development