Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-741

TestHFlush test doesn't seek() past previously written part of the file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As a part of the test scenario a file is being written, 10th of the total length in a time. Then a reader is opened to read what has been just written and hflush'ed. However, it always starts reading from the 0 position of the file and doesn't seek to the start of the portion written last.

      1. HDFS-741.patch
        0.7 kB
        Konstantin Boudnik
      2. HDFS-741.patch
        0.8 kB
        Konstantin Boudnik
      3. HDFS-741.patch
        2 kB
        Konstantin Boudnik
      4. h741_20091112.patch
        3 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Hide
          Konstantin Boudnik added a comment -

          This fixes the seek problem.

          However, two testcases (writing cross block and checksum boundaries) are failing.
          From the sizes of a block and checksum is seems that I can't seek over block boundary and it might be a limitation of the test... I'm not sure....

          Show
          Konstantin Boudnik added a comment - This fixes the seek problem. However, two testcases (writing cross block and checksum boundaries) are failing. From the sizes of a block and checksum is seems that I can't seek over block boundary and it might be a limitation of the test... I'm not sure....
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Attachment HDFS-741.patch [ 12423514 ]
          Konstantin Boudnik made changes -
          Link This issue blocks HDFS-706 [ HDFS-706 ]
          Hide
          Hairong Kuang added a comment -

          Cos, could you please try verifying the whole file instead of the newly hflushed data. This way we can identify if this is a bug of seek or a bug of hflush.

          Show
          Hairong Kuang added a comment - Cos, could you please try verifying the whole file instead of the newly hflushed data. This way we can identify if this is a bug of seek or a bug of hflush.
          Hide
          Konstantin Boudnik added a comment -

          With this one it is much more apparent that seek() is failing.

          Show
          Konstantin Boudnik added a comment - With this one it is much more apparent that seek() is failing.
          Konstantin Boudnik made changes -
          Attachment HDFS-741.patch [ 12423530 ]
          Hide
          Konstantin Boudnik added a comment -

          I have verified it: the whole file reads Ok. I've simply commented out checkData(..) call inside of the loop and tests start passing. Seems to me as a seek()} problem.

          Show
          Konstantin Boudnik added a comment - I have verified it: the whole file reads Ok. I've simply commented out checkData(..) call inside of the loop and tests start passing. Seems to me as a seek() } problem.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          +        int readBytes = is.read(toRead, 0, tenth);
          +        assertEquals("Should've get more bytes", tenth, readBytes);
          

          Why expecting (tenth == readBytes)? java.io.DataInputStream.read(..) may read some number of bytes up to the given length but not necessary all the bytes, i.e. it is okay if we have (0 < readBytes <= tenth).

          Show
          Tsz Wo Nicholas Sze added a comment - + int readBytes = is.read(toRead, 0, tenth); + assertEquals( "Should've get more bytes" , tenth, readBytes); Why expecting (tenth == readBytes)? java.io.DataInputStream.read(..) may read some number of bytes up to the given length but not necessary all the bytes, i.e. it is okay if we have (0 < readBytes <= tenth).
          Hide
          Konstantin Boudnik added a comment -

          It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.

          Show
          Konstantin Boudnik added a comment - It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.
          Konstantin Boudnik made changes -
          Attachment HDFS-741.patch [ 12424767 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that.

          Hi Cos, checkData(..) should check only byte 0 to (readBytes-1). h741_20091112.patch works in my machine.

          h741_20091112.patch: changed checkData(..).

          Show
          Tsz Wo Nicholas Sze added a comment - > It seems that you're right. However, the point is still valid although is slightly different: I'm able to read >0 bytes after seek. However, they content is filled with '0' rather then with something the test is writing into the file. Slightly modified version of the test to demonstrate just that. Hi Cos, checkData(..) should check only byte 0 to (readBytes-1). h741_20091112.patch works in my machine. h741_20091112.patch: changed checkData(..).
          Tsz Wo Nicholas Sze made changes -
          Attachment h741_20091112.patch [ 12424785 ]
          Hide
          Konstantin Boudnik added a comment -

          Yes, you're right. My bad. Thanks for correcting it.

          Show
          Konstantin Boudnik added a comment - Yes, you're right. My bad. Thanks for correcting it.
          Hide
          Konstantin Boudnik added a comment -

          Latest patch by Nicholas seems to be fixing the issue

          Show
          Konstantin Boudnik added a comment - Latest patch by Nicholas seems to be fixing the issue
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424785/h741_20091112.patch
          against trunk revision 835534.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424785/h741_20091112.patch against trunk revision 835534. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/75/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good.

          Test failures are expected and are addressed by HDFS-706. As soon as HDFS-706 is reviewed, I'm going to commit this modification, run test-patch for HDFS-706 and commit it as well to minimize time of broken test-patch.

          Show
          Konstantin Boudnik added a comment - +1 patch looks good. Test failures are expected and are addressed by HDFS-706 . As soon as HDFS-706 is reviewed, I'm going to commit this modification, run test-patch for HDFS-706 and commit it as well to minimize time of broken test-patch.
          Hide
          Konstantin Boudnik added a comment -

          This patch has to be merged to branch 0.21.
          Few testcases in TestFiHFlush will start failing after this patch. HDFS-706 will fix them shortly.

          I've just committed this.

          Show
          Konstantin Boudnik added a comment - This patch has to be merged to branch 0.21. Few testcases in TestFiHFlush will start failing after this patch. HDFS-706 will fix them shortly. I've just committed this.
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #113 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/113/)
          . TestHFlush test doesn't seek() past previously written part of the file. Contributed by Konstantin Boudnik

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #113 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/113/ ) . TestHFlush test doesn't seek() past previously written part of the file. Contributed by Konstantin Boudnik
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #145 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/145/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #145 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/145/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #118 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/118/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #118 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/118/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/ )
          Tom White made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          14d 23h 39m 1 Konstantin Boudnik 12/Nov/09 23:36
          Patch Available Patch Available Resolved Resolved
          3d 22h 52m 1 Konstantin Boudnik 16/Nov/09 22:29
          Resolved Resolved Closed Closed
          280d 22h 20m 1 Tom White 24/Aug/10 21:50

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development