Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).

      1. MAPREDUCE-2156.2.patch
        40 kB
        Patrick Kling
      2. MAPREDUCE-2156.3.patch
        41 kB
        Patrick Kling
      3. MAPREDUCE-2156.patch
        41 kB
        Patrick Kling

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/)
          MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/ ) MAPREDUCE-2156 . Raid-aware FSCK. (Patrick Kling via dhruba)
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Patrick!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Patrick!
          Hide
          Patrick Kling added a comment -

          Updated README.

          Show
          Patrick Kling added a comment - Updated README.
          Hide
          Ramkumar Vadali added a comment -

          +1, looks good.

          Show
          Ramkumar Vadali added a comment - +1, looks good.
          Hide
          Patrick Kling added a comment -

          Removed commented-out code.

          ParityBlockInfo is no longer needed since we now use HarFileSystem.getFileBlockLocations() from MAPREDUCE-1752.

          Show
          Patrick Kling added a comment - Removed commented-out code. ParityBlockInfo is no longer needed since we now use HarFileSystem.getFileBlockLocations() from MAPREDUCE-1752 .
          Hide
          Patrick Kling added a comment -

          See review board: https://reviews.apache.org/r/132/

          ant test-patch output:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Patrick Kling added a comment - See review board: https://reviews.apache.org/r/132/ ant test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development