Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It turned out that there's no test for block report functionality. The one would be extremely valuable.

      1. HDFS-551.patch
        8 kB
        Konstantin Boudnik
      2. HDFS-551.patch
        9 kB
        Konstantin Boudnik
      3. BlockReportTestPlan.html
        10 kB
        Konstantin Boudnik
      4. BlockReportTestPlan.html
        10 kB
        Konstantin Boudnik
      5. HDFS-551.patch
        9 kB
        Konstantin Boudnik
      6. HDFS-551.patch
        10 kB
        Konstantin Boudnik
      7. HDFS-551.patch
        10 kB
        Konstantin Boudnik
      8. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      9. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      10. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      11. BlockReportTestPlan.html
        11 kB
        Konstantin Boudnik
      12. BlockReportTestPlan.html
        9 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Last three test cases were removed for they are essentially re-enforce the behavior of BlockReport_09

          Show
          Konstantin Boudnik added a comment - Last three test cases were removed for they are essentially re-enforce the behavior of BlockReport_09
          Hide
          Konstantin Boudnik added a comment -

          _08 test case is updated according to the specs

          Show
          Konstantin Boudnik added a comment - _08 test case is updated according to the specs
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #74 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/74/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #74 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/74/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #4 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/4/)
          . Create new functional test for a block report. Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #4 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/4/ ) . Create new functional test for a block report. Contributed by Konstantin Boudnik.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/15/)
          . Create new functional test for a block report. Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/15/ ) . Create new functional test for a block report. Contributed by Konstantin Boudnik.
          Hide
          Konstantin Boudnik added a comment -

          The JIRA was already created (HDFS-591) and I have linked it to this one. Thanks for noticing and for the commit!

          Show
          Konstantin Boudnik added a comment - The JIRA was already created ( HDFS-591 ) and I have linked it to this one. Thanks for noticing and for the commit!
          Hide
          Hairong Kuang added a comment -

          I committed this to the trunk. Thanks Cos!

          The first test is not an expected behavior. Cos, could you please file a bug on this?

          Show
          Hairong Kuang added a comment - I committed this to the trunk. Thanks Cos! The first test is not an expected behavior. Cos, could you please file a bug on this?
          Hide
          Konstantin Boudnik added a comment -

          The failing test is unrelated to the patch: it has been failing intermittently for some time now.

          Show
          Konstantin Boudnik added a comment - The failing test is unrelated to the patch: it has been failing intermittently for some time now.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418462/HDFS-551.patch
          against trunk revision 810631.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418462/HDFS-551.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          The patch seems to be ready for prime-time

          Show
          Konstantin Boudnik added a comment - The patch seems to be ready for prime-time
          Hide
          Konstantin Boudnik added a comment -

          Non-principal change to improve the code's readability.

          Show
          Konstantin Boudnik added a comment - Non-principal change to improve the code's readability.
          Hide
          Hairong Kuang added a comment -

          +1

          Show
          Hairong Kuang added a comment - +1
          Hide
          Konstantin Boudnik added a comment -

          Hairong's comments are addressed. Thanks for the review.
          Also, I'm gonna open new JIRA for the incorrect NN's behavior and move this one into the append branch

          Show
          Konstantin Boudnik added a comment - Hairong's comments are addressed. Thanks for the review. Also, I'm gonna open new JIRA for the incorrect NN's behavior and move this one into the append branch
          Hide
          Hairong Kuang added a comment -

          DFSTestUtils has a set of utility functions to support file creation etc. You may take a look so you do not need to invent your own.

          Show
          Hairong Kuang added a comment - DFSTestUtils has a set of utility functions to support file creation etc. You may take a look so you do not need to invent your own.
          Hide
          Hairong Kuang added a comment -

          For the messWithBlocksLen() test, If a datanode reports a replica with the different length from the Complete block size in NN, NN should mark this replica as corrupt. NN should not update the block's length.

          Please also remove all the commented statements in the patch.

          Show
          Hairong Kuang added a comment - For the messWithBlocksLen() test, If a datanode reports a replica with the different length from the Complete block size in NN, NN should mark this replica as corrupt. NN should not update the block's length. Please also remove all the commented statements in the patch.
          Hide
          Konstantin Boudnik added a comment -

          JavaDocs are added for the test cases.

          Show
          Konstantin Boudnik added a comment - JavaDocs are added for the test cases.
          Hide
          Konstantin Boudnik added a comment -

          All verifications are working as expected now.

          Show
          Konstantin Boudnik added a comment - All verifications are working as expected now.
          Hide
          Konstantin Boudnik added a comment -

          I can see that BlockReport does the right thing and blocks deleted on the DN side are removed from the name system as well.

          However, I still can't programmaticaly verity that NN's block report does everything correctly.

          Show
          Konstantin Boudnik added a comment - I can see that BlockReport does the right thing and blocks deleted on the DN side are removed from the name system as well. However, I still can't programmaticaly verity that NN's block report does everything correctly.
          Hide
          Konstantin Boudnik added a comment -

          The verification of NN's block report works now. however one of the test cases can't get correct number of under-replicated blocks yet.

          Show
          Konstantin Boudnik added a comment - The verification of NN's block report works now. however one of the test cases can't get correct number of under-replicated blocks yet.
          Hide
          Konstantin Boudnik added a comment -

          Initial version of Block Report test plan as per new design.

          Show
          Konstantin Boudnik added a comment - Initial version of Block Report test plan as per new design.
          Hide
          Konstantin Boudnik added a comment -

          Initial version of Block Report test plan as per new design.

          Show
          Konstantin Boudnik added a comment - Initial version of Block Report test plan as per new design.
          Hide
          Konstantin Boudnik added a comment -

          This JIRA includes some test cases for existing block report feature.
          Eventually, it will be evolving to cover new design (post append) of block report functionality.

          Show
          Konstantin Boudnik added a comment - This JIRA includes some test cases for existing block report feature. Eventually, it will be evolving to cover new design (post append) of block report functionality.
          Hide
          Konstantin Boudnik added a comment -

          80 characters rule is applied

          Show
          Konstantin Boudnik added a comment - 80 characters rule is applied
          Hide
          Konstantin Boudnik added a comment -

          Two initial test cases. Not working as expected yet.

          Show
          Konstantin Boudnik added a comment - Two initial test cases. Not working as expected yet.

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development