Hadoop Common
  1. Hadoop Common
  2. HADOOP-4277

Checksum verification is disabled for LocalFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.3, 0.18.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Checksum verification was mistakenly disabled for LocalFileSystem.

      Description

      Since HADOOP-2063 checksum verification is disabled on ChecksumFileSystem used by LocalFS and others. HDFS is not affected.

      We should add test case that corrupts LocalFS file in TestFSInputChecker.java.

      1. HADOOP-4277.patch
        4 kB
        Raghu Angadi
      2. HADOOP-4277-branch-17.patch
        4 kB
        Raghu Angadi
      3. HADOOP-4277-branch-18.patch
        4 kB
        Raghu Angadi
      4. HADOOP-4277.patch
        4 kB
        Raghu Angadi
      5. HADOOP-4277-branch-17.patch
        4 kB
        Raghu Angadi
      6. HADOOP-4277-branch-18.patch
        4 kB
        Raghu Angadi
      7. HADOOP-4277.patch
        4 kB
        Raghu Angadi
      8. HADOOP-4277.patch
        6 kB
        Raghu Angadi
      9. HADOOP-4277-branch-17.patch
        5 kB
        Raghu Angadi
      10. HADOOP-4277-branch-18.patch
        4 kB
        Raghu Angadi
      11. HADOOP-4277.patch
        5 kB
        Raghu Angadi
      12. HADOOP-4277-branch-18-correction.patch
        1 kB
        Raghu Angadi

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          The patch attached for 0.18 above is incomplete. I committed attached HADOOP-4277-branch-18-correction.patch in addition.

          Show
          Raghu Angadi added a comment - The patch attached for 0.18 above is incomplete. I committed attached HADOOP-4277 -branch-18-correction.patch in addition.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #616 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/616/ )
          Hide
          Raghu Angadi added a comment -

          I just committed this.

          'ant test-patch' output :

               [exec] +1 overall.
          
               [exec]     +1 @author.  The patch does not contain any @author tags.
          
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
          
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          
          
          Show
          Raghu Angadi added a comment - I just committed this. 'ant test-patch' output : [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          Hide
          Raghu Angadi added a comment -

          I am planning to commit these before Hudson since 0.17.3 is waiting on this.

          Show
          Raghu Angadi added a comment - I am planning to commit these before Hudson since 0.17.3 is waiting on this.
          Hide
          Raghu Angadi added a comment -

          This fix triggered two existing bugs :

          • TestDFSShell.testPut() : affected by left over crc files from previous tests. Fixed in this patch.
          • TestLocalFileSystem.testAppend() : filed HADOOP-4292 and disabled the test here.

          No other files have changed.

          Show
          Raghu Angadi added a comment - This fix triggered two existing bugs : TestDFSShell.testPut() : affected by left over crc files from previous tests. Fixed in this patch. TestLocalFileSystem.testAppend() : filed HADOOP-4292 and disabled the test here. No other files have changed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12390958/HADOOP-4277.patch
          against trunk revision 699119.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390958/HADOOP-4277.patch against trunk revision 699119. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3373/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          Test creates the files under test.build.data directory rather than in current working directory.

          Show
          Raghu Angadi added a comment - Test creates the files under test.build.data directory rather than in current working directory.
          Hide
          Raghu Angadi added a comment -

          Resubmitting the patch for trunk for Hudson.

          Show
          Raghu Angadi added a comment - Resubmitting the patch for trunk for Hudson.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 Patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 Patch looks good.
          Hide
          Raghu Angadi added a comment -

          Patch for trunk. TestFSInputChecker add two cases where crc file or data is corrutped. I will attach patches for other branches later.

          Show
          Raghu Angadi added a comment - Patch for trunk. TestFSInputChecker add two cases where crc file or data is corrutped. I will attach patches for other branches later.

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development