Hadoop Common
  1. Hadoop Common
  2. HADOOP-6177

FSInputChecker.getPos() would return position greater than the file size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

      1. hadoop-6177-testcase.patch
        2 kB
        Hong Tang
      2. hadoop-6177-2090806.patch
        3 kB
        Hong Tang

        Activity

        Hide
        Hong Tang added a comment -

        The patch file modifies existing test cases to check that after reading through the file, the final offset would be the same as the file offset. On my local machine, it yields:

        % ant test-core -Dtestcase=TestChecksumFileSystem
        run-test-core:
           [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data
            [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data
           [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs
            [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs
            [junit] Running org.apache.hadoop.fs.TestChecksumFileSystem
            [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 0.59 sec
            [junit] Test org.apache.hadoop.fs.TestChecksumFileSystem FAILED
        % tail build/test/TEST-org.apache.hadoop.fs.TestChecksumFileSystem.txt
        ------------- ---------------- ---------------
        
        Testcase: testgetChecksumLength took 0.196 sec
        Testcase: testVerifyChecksum took 0.373 sec
                FAILED
        expected:<7> but was:<8>
        junit.framework.AssertionFailedError: expected:<7> but was:<8>
                at org.apache.hadoop.fs.TestLocalFileSystem.readFile(TestLocalFileSystem.java:45)
                at org.apache.hadoop.fs.TestChecksumFileSystem.testVerifyChecksum(TestChecksumFileSystem.java:57)
        
        Show
        Hong Tang added a comment - The patch file modifies existing test cases to check that after reading through the file, the final offset would be the same as the file offset. On my local machine, it yields: % ant test-core -Dtestcase=TestChecksumFileSystem run-test-core: [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs [junit] Running org.apache.hadoop.fs.TestChecksumFileSystem [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 0.59 sec [junit] Test org.apache.hadoop.fs.TestChecksumFileSystem FAILED % tail build/test/TEST-org.apache.hadoop.fs.TestChecksumFileSystem.txt ------------- ---------------- --------------- Testcase: testgetChecksumLength took 0.196 sec Testcase: testVerifyChecksum took 0.373 sec FAILED expected:<7> but was:<8> junit.framework.AssertionFailedError: expected:<7> but was:<8> at org.apache.hadoop.fs.TestLocalFileSystem.readFile(TestLocalFileSystem.java:45) at org.apache.hadoop.fs.TestChecksumFileSystem.testVerifyChecksum(TestChecksumFileSystem.java:57)
        Hide
        Hong Tang added a comment -

        The root cause is because readChecksumChunk returns -1 when the fs input stream reaches the end of the file. And in fill(), this leads to count == -1 (FSInputChecker.java:176). The fix is rather simple, adding the following line at the end of fill():

             count = readChecksumChunk(buf, 0, buf.length);
        +    if (count < 0) count = 0; 
        
        Show
        Hong Tang added a comment - The root cause is because readChecksumChunk returns -1 when the fs input stream reaches the end of the file. And in fill(), this leads to count == -1 (FSInputChecker.java:176). The fix is rather simple, adding the following line at the end of fill(): count = readChecksumChunk(buf, 0, buf.length); + if (count < 0) count = 0;
        Hide
        Hong Tang added a comment -

        Patch contains both modified test cases and the one line fix.

        Show
        Hong Tang added a comment - Patch contains both modified test cases and the one line fix.
        Hide
        Hong Tang added a comment -

        ant test and ant test-patch passed on my local machine:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 6 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Hong Tang added a comment - ant test and ant test-patch passed on my local machine: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12415815/hadoop-6177-2090806.patch
        against trunk revision 800919.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415815/hadoop-6177-2090806.patch against trunk revision 800919. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/console This message is automatically generated.
        Hide
        Hairong Kuang added a comment -

        Good catch, Hong!
        +1 The patch looks good.

        Show
        Hairong Kuang added a comment - Good catch, Hong! +1 The patch looks good.
        Hide
        Hairong Kuang added a comment -

        I just committed this. Thanks Hong!

        Show
        Hairong Kuang added a comment - I just committed this. Thanks Hong!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #52 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/52/)
        . FSInputChecker.getPos() would return position greater than the file size. Contributed by Hong Tang.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #52 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/52/ ) . FSInputChecker.getPos() would return position greater than the file size. Contributed by Hong Tang.

          People

          • Assignee:
            Hong Tang
            Reporter:
            Hong Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development