Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-728

Create a comprehensive functional test for append

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This test aims to do
      1. create a file of len1;
      2. reopen the file for append;
      3. write len2 bytes to the file and hflush;
      4. write len3 bytes to the file and close the file;
      5. validate the content of the file.

      Len1 ranges from [0, 2*BLOCK_SIZE+1], len2 ranges from [0, BLOCK_SIZE+1], and len3 ranges from [0, BLOCK_SIZE+1]. The test tries all combination of len1, len2, and len3. To minimize the running time, bytes per checksum is set to be 4 bytes, each packet size is set to be bytes per checksum, and each block contains 2 packets.

      1. appendTest1.patch
        5 kB
        Hairong Kuang
      2. HDFS-728.patch
        5 kB
        Konstantin Boudnik
      3. appendTest.patch
        4 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          Here is a patch that implements the above idea.

          Show
          Hairong Kuang added a comment - Here is a patch that implements the above idea.
          Hide
          Konstantin Boudnik added a comment -

          There was a JIRA for append tests (HDFS-522). I'll close it as duplicate of this one.

          Show
          Konstantin Boudnik added a comment - There was a JIRA for append tests ( HDFS-522 ). I'll close it as duplicate of this one.
          Hide
          Konstantin Boudnik added a comment -

          Actually, I'm confused - HDFS-522 is about stress tests for append. Please disregard the comment above

          Show
          Konstantin Boudnik added a comment - Actually, I'm confused - HDFS-522 is about stress tests for append. Please disregard the comment above
          Hide
          Konstantin Boudnik added a comment -

          I've ran the test by the means of Ant build and it hands after 15 minutes. Apparently it happens after JUnit timeout is reached.

          Show
          Konstantin Boudnik added a comment - I've ran the test by the means of Ant build and it hands after 15 minutes. Apparently it happens after JUnit timeout is reached.
          Hide
          Hairong Kuang added a comment -

          This test is not intended as a junit test. Instead, it should run as an application from the command line.

          Show
          Hairong Kuang added a comment - This test is not intended as a junit test. Instead, it should run as an application from the command line.
          Hide
          Konstantin Boudnik added a comment -

          Right, I've understood this. My question is: "How do we suppose to run it in Hudson then?"

          It has to be executed by Hudson on a regular basis if it suppose to be a part of standard development cycle. Shell it be moved to a special load (or stressful) test suite? It might be an open discussion for this. However, there's no way to run a command line application as a part of CI build/test cycle.

          Show
          Konstantin Boudnik added a comment - Right, I've understood this. My question is: "How do we suppose to run it in Hudson then?" It has to be executed by Hudson on a regular basis if it suppose to be a part of standard development cycle. Shell it be moved to a special load (or stressful) test suite? It might be an open discussion for this. However, there's no way to run a command line application as a part of CI build/test cycle.
          Hide
          Konstantin Boudnik added a comment -

          What I can offer on another hand is to run this 'comprehensive' test under JUnit control. However, in order to avoid timeout issues it needs to be split in a few smaller sub-tests.
          And the mini-cluster is needed to be brought up only one per class in this case.

          Show
          Konstantin Boudnik added a comment - What I can offer on another hand is to run this 'comprehensive' test under JUnit control. However, in order to avoid timeout issues it needs to be split in a few smaller sub-tests. And the mini-cluster is needed to be brought up only one per class in this case.
          Hide
          Nigel Daley added a comment -

          Hairong, please write this as a Junit test. You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). We should then file a Jira to add a functional test target to Ant that can pick up this and other similar tests.

          Show
          Nigel Daley added a comment - Hairong, please write this as a Junit test. You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). We should then file a Jira to add a functional test target to Ant that can pick up this and other similar tests.
          Hide
          Konstantin Boudnik added a comment -

          Merging together last two comments and adding some thoughts on top:

          • no split of the test case is required then. So, my patch could be modified slightly and seems to be good to go.
          • this is clearly a functional test, so it make sense to create a separate directory structure like test/src/functional
          • for this new target a different timeout settings would be required
          Show
          Konstantin Boudnik added a comment - Merging together last two comments and adding some thoughts on top: no split of the test case is required then. So, my patch could be modified slightly and seems to be good to go. this is clearly a functional test, so it make sense to create a separate directory structure like test/src/functional for this new target a different timeout settings would be required
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). ...

          We have already moved to junit 4 and so we are using the @Test annotation. The name of the test does not matter.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). ... We have already moved to junit 4 and so we are using the @Test annotation. The name of the test does not matter.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          >> ... You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). ...
          >
          > We have already moved to junit 4 and so we are using the @Test annotation. The name of the test does not matter.

          It is not true. Only the method name does not matter but not the class name.

          Show
          Tsz Wo Nicholas Sze added a comment - >> ... You can name the test appropriately so that it doesn't get picked up by the test target (simply don't start the test class name with the word Test). ... > > We have already moved to junit 4 and so we are using the @Test annotation. The name of the test does not matter. It is not true. Only the method name does not matter but not the class name.
          Hide
          Hairong Kuang added a comment -

          For this issue, I plan to take Nigel's suggestion for now.

          > this is clearly a functional test, so it make sense to create a separate directory structure like test/src/functional.
          +1. This is the way to go! We also need a new target in build.xml. Shall we do it in a different jira?

          Show
          Hairong Kuang added a comment - For this issue, I plan to take Nigel's suggestion for now. > this is clearly a functional test, so it make sense to create a separate directory structure like test/src/functional. +1. This is the way to go! We also need a new target in build.xml. Shall we do it in a different jira?
          Hide
          Konstantin Boudnik added a comment -

          And, the target creation has to be done as a separate JIRA's patch

          Show
          Konstantin Boudnik added a comment - And, the target creation has to be done as a separate JIRA's patch
          Hide
          Hairong Kuang added a comment -

          This patch makes the test to be a unit test and changes the class name not to start with Test. Part of the patch came from Cos' patch. Thanks Cos!

          Show
          Hairong Kuang added a comment - This patch makes the test to be a unit test and changes the class name not to start with Test. Part of the patch came from Cos' patch. Thanks Cos!
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good

          Show
          Konstantin Boudnik added a comment - +1 patch looks good
          Hide
          Hairong Kuang added a comment -

          I've committed this!

          Show
          Hairong Kuang added a comment - I've committed this!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/85/)
          . Create a comprehensive functional test for append. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/85/ ) . Create a comprehensive functional test for append. Contributed by Hairong Kuang.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #122 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/122/)
          . Create a comprehensive functional test for append. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #122 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/122/ ) . Create a comprehensive functional test for append. Contributed by Hairong Kuang.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #65 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/65/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #65 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/65/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/ )

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development