Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-741

TestHFlush test doesn't seek() past previously written part of the file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As a part of the test scenario a file is being written, 10th of the total length in a time. Then a reader is opened to read what has been just written and hflush'ed. However, it always starts reading from the 0 position of the file and doesn't seek to the start of the portion written last.

      1. h741_20091112.patch
        3 kB
        Tsz Wo Nicholas Sze
      2. HDFS-741.patch
        2 kB
        Konstantin Boudnik
      3. HDFS-741.patch
        0.8 kB
        Konstantin Boudnik
      4. HDFS-741.patch
        0.7 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          This fixes the seek problem.

          However, two testcases (writing cross block and checksum boundaries) are failing.
          From the sizes of a block and checksum is seems that I can't seek over block boundary and it might be a limitation of the test... I'm not sure....

          Show
          Konstantin Boudnik added a comment - This fixes the seek problem. However, two testcases (writing cross block and checksum boundaries) are failing. From the sizes of a block and checksum is seems that I can't seek over block boundary and it might be a limitation of the test... I'm not sure....
          Hide
          Hairong Kuang added a comment -

          Cos, could you please try verifying the whole file instead of the newly hflushed data. This way we can identify if this is a bug of seek or a bug of hflush.

          Show
          Hairong Kuang added a comment - Cos, could you please try verifying the whole file instead of the newly hflushed data. This way we can identify if this is a bug of seek or a bug of hflush.
          Hide
          Konstantin Boudnik added a comment -

          With this one it is much more apparent that seek() is failing.

          Show
          Konstantin Boudnik added a comment - With this one it is much more apparent that seek() is failing.
          Hide
          Konstantin Boudnik added a comment -

          I have verified it: the whole file reads Ok. I've simply commented out checkData(..) call inside of the loop and tests start passing. Seems to me as a seek()} problem.

          Show
          Konstantin Boudnik added a comment - I have verified it: the whole file reads Ok. I've simply commented out checkData(..) call inside of the loop and tests start passing. Seems to me as a seek() } problem.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          +        int readBytes = is.read(toRead, 0, tenth);
          +        assertEquals("Should've get more bytes", tenth, readBytes);
          

          Why expecting (tenth == readBytes)? java.io.DataInputStream.read(..) may read some number of bytes up to the given length but not necessary all the bytes, i.e. it is okay if we have (0 < readBytes <= tenth).

          Show
          Tsz Wo Nicholas Sze added a comment - + int readBytes = is.read(toRead, 0, tenth); + assertEquals( "Should've get more bytes" , tenth, readBytes); Why expecting (tenth == readBytes)? java.io.DataInputStream.read(..) may read some number of bytes up to the given length but not necessary all the bytes, i.e. it is okay if we have (0 < readBytes <= tenth).
          Hide
          Konstantin Boudnik added a comment -

          It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.

          Show
          Konstantin Boudnik added a comment - It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.

          Hi Cos, checkData(..) should check only byte 0 to (readBytes-1). h741_20091112.patch works in my machine.

          h741_20091112.patch: changed checkData(..).

          Show
          Tsz Wo Nicholas Sze added a comment - > It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that. Hi Cos, checkData(..) should check only byte 0 to (readBytes-1). h741_20091112.patch works in my machine. h741_20091112.patch: changed checkData(..).
          Hide
          Konstantin Boudnik added a comment -

          Yes, you're right. My bad. Thanks for correcting it.

          Show
          Konstantin Boudnik added a comment - Yes, you're right. My bad. Thanks for correcting it.
          Hide
          Konstantin Boudnik added a comment -

          Latest patch by Nicholas seems to be fixing the issue

          Show
          Konstantin Boudnik added a comment - Latest patch by Nicholas seems to be fixing the issue
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424785/h741_20091112.patch
          against trunk revision 835534.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424785/h741_20091112.patch against trunk revision 835534. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good.

          Test failures are expected and are addressed by HDFS-706. As soon as HDFS-706 is reviewed, I'm going to commit this modification, run test-patch for HDFS-706 and commit it as well to minimize time of broken test-patch.

          Show
          Konstantin Boudnik added a comment - +1 patch looks good. Test failures are expected and are addressed by HDFS-706 . As soon as HDFS-706 is reviewed, I'm going to commit this modification, run test-patch for HDFS-706 and commit it as well to minimize time of broken test-patch.
          Hide
          Konstantin Boudnik added a comment -

          This patch has to be merged to branch 0.21.
          Few testcases in TestFiHFlush will start failing after this patch. HDFS-706 will fix them shortly.

          I've just committed this.

          Show
          Konstantin Boudnik added a comment - This patch has to be merged to branch 0.21. Few testcases in TestFiHFlush will start failing after this patch. HDFS-706 will fix them shortly. I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #113 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/113/)
          . TestHFlush test doesn't seek() past previously written part of the file. Contributed by Konstantin Boudnik

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #113 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/113/ ) . TestHFlush test doesn't seek() past previously written part of the file. Contributed by Konstantin Boudnik
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #145 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/145/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #145 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/145/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #118 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/118/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #118 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/118/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/ )

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development