Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It turned out that there's no test for block report functionality. The one would be extremely valuable.

      1. HDFS-551.patch
        8 kB
        Konstantin Boudnik
      2. HDFS-551.patch
        9 kB
        Konstantin Boudnik
      3. BlockReportTestPlan.html
        10 kB
        Konstantin Boudnik
      4. BlockReportTestPlan.html
        10 kB
        Konstantin Boudnik
      5. HDFS-551.patch
        9 kB
        Konstantin Boudnik
      6. HDFS-551.patch
        10 kB
        Konstantin Boudnik
      7. HDFS-551.patch
        10 kB
        Konstantin Boudnik
      8. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      9. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      10. HDFS-551.patch
        11 kB
        Konstantin Boudnik
      11. BlockReportTestPlan.html
        11 kB
        Konstantin Boudnik
      12. BlockReportTestPlan.html
        9 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Assignee Konstantin Boudnik [ cos ]
          Konstantin Boudnik made changes -
          Component/s test [ 12312916 ]
          Hide
          Konstantin Boudnik added a comment -

          Two initial test cases. Not working as expected yet.

          Show
          Konstantin Boudnik added a comment - Two initial test cases. Not working as expected yet.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12417340 ]
          Hide
          Konstantin Boudnik added a comment -

          80 characters rule is applied

          Show
          Konstantin Boudnik added a comment - 80 characters rule is applied
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12417342 ]
          Konstantin Boudnik made changes -
          Link This issue is part of HDFS-265 [ HDFS-265 ]
          Hide
          Konstantin Boudnik added a comment -

          This JIRA includes some test cases for existing block report feature.
          Eventually, it will be evolving to cover new design (post append) of block report functionality.

          Show
          Konstantin Boudnik added a comment - This JIRA includes some test cases for existing block report feature. Eventually, it will be evolving to cover new design (post append) of block report functionality.
          Hide
          Konstantin Boudnik added a comment -

          Initial version of Block Report test plan as per new design.

          Show
          Konstantin Boudnik added a comment - Initial version of Block Report test plan as per new design.
          Konstantin Boudnik made changes -
          Attachment BlockReportTestPlan.html [ 12417512 ]
          Hide
          Konstantin Boudnik added a comment -

          Initial version of Block Report test plan as per new design.

          Show
          Konstantin Boudnik added a comment - Initial version of Block Report test plan as per new design.
          Konstantin Boudnik made changes -
          Attachment BlockReportTestPlan.html [ 12417513 ]
          Hide
          Konstantin Boudnik added a comment -

          The verification of NN's block report works now. however one of the test cases can't get correct number of under-replicated blocks yet.

          Show
          Konstantin Boudnik added a comment - The verification of NN's block report works now. however one of the test cases can't get correct number of under-replicated blocks yet.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12417916 ]
          Hide
          Konstantin Boudnik added a comment -

          I can see that BlockReport does the right thing and blocks deleted on the DN side are removed from the name system as well.

          However, I still can't programmaticaly verity that NN's block report does everything correctly.

          Show
          Konstantin Boudnik added a comment - I can see that BlockReport does the right thing and blocks deleted on the DN side are removed from the name system as well. However, I still can't programmaticaly verity that NN's block report does everything correctly.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12417924 ]
          Hide
          Konstantin Boudnik added a comment -

          All verifications are working as expected now.

          Show
          Konstantin Boudnik added a comment - All verifications are working as expected now.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12418201 ]
          Hide
          Konstantin Boudnik added a comment -

          JavaDocs are added for the test cases.

          Show
          Konstantin Boudnik added a comment - JavaDocs are added for the test cases.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12418303 ]
          Hide
          Hairong Kuang added a comment -

          For the messWithBlocksLen() test, If a datanode reports a replica with the different length from the Complete block size in NN, NN should mark this replica as corrupt. NN should not update the block's length.

          Please also remove all the commented statements in the patch.

          Show
          Hairong Kuang added a comment - For the messWithBlocksLen() test, If a datanode reports a replica with the different length from the Complete block size in NN, NN should mark this replica as corrupt. NN should not update the block's length. Please also remove all the commented statements in the patch.
          Hide
          Hairong Kuang added a comment -

          DFSTestUtils has a set of utility functions to support file creation etc. You may take a look so you do not need to invent your own.

          Show
          Hairong Kuang added a comment - DFSTestUtils has a set of utility functions to support file creation etc. You may take a look so you do not need to invent your own.
          Hide
          Konstantin Boudnik added a comment -

          Hairong's comments are addressed. Thanks for the review.
          Also, I'm gonna open new JIRA for the incorrect NN's behavior and move this one into the append branch

          Show
          Konstantin Boudnik added a comment - Hairong's comments are addressed. Thanks for the review. Also, I'm gonna open new JIRA for the incorrect NN's behavior and move this one into the append branch
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12418437 ]
          Konstantin Boudnik made changes -
          Affects Version/s Append Branch [ 12314142 ]
          Hide
          Hairong Kuang added a comment -

          +1

          Show
          Hairong Kuang added a comment - +1
          Konstantin Boudnik made changes -
          Parent HDFS-265 [ 12423739 ]
          Issue Type Test [ 6 ] Sub-task [ 7 ]
          Konstantin Boudnik made changes -
          Link This issue is part of HDFS-265 [ HDFS-265 ]
          Hide
          Konstantin Boudnik added a comment -

          Non-principal change to improve the code's readability.

          Show
          Konstantin Boudnik added a comment - Non-principal change to improve the code's readability.
          Konstantin Boudnik made changes -
          Attachment HDFS-551.patch [ 12418462 ]
          Hide
          Konstantin Boudnik added a comment -

          The patch seems to be ready for prime-time

          Show
          Konstantin Boudnik added a comment - The patch seems to be ready for prime-time
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.21.0 [ 12314046 ]
          Affects Version/s Append Branch [ 12314142 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418462/HDFS-551.patch
          against trunk revision 810631.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418462/HDFS-551.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/9/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          The failing test is unrelated to the patch: it has been failing intermittently for some time now.

          Show
          Konstantin Boudnik added a comment - The failing test is unrelated to the patch: it has been failing intermittently for some time now.
          Hide
          Hairong Kuang added a comment -

          I committed this to the trunk. Thanks Cos!

          The first test is not an expected behavior. Cos, could you please file a bug on this?

          Show
          Hairong Kuang added a comment - I committed this to the trunk. Thanks Cos! The first test is not an expected behavior. Cos, could you please file a bug on this?
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314046 ]
          Resolution Fixed [ 1 ]
          Konstantin Boudnik made changes -
          Link This issue depends upon HDFS-591 [ HDFS-591 ]
          Hide
          Konstantin Boudnik added a comment -

          The JIRA was already created (HDFS-591) and I have linked it to this one. Thanks for noticing and for the commit!

          Show
          Konstantin Boudnik added a comment - The JIRA was already created ( HDFS-591 ) and I have linked it to this one. Thanks for noticing and for the commit!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/15/)
          . Create new functional test for a block report. Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/15/ ) . Create new functional test for a block report. Contributed by Konstantin Boudnik.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #4 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/4/)
          . Create new functional test for a block report. Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #4 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/4/ ) . Create new functional test for a block report. Contributed by Konstantin Boudnik.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #74 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/74/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #74 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/74/ )
          Konstantin Shvachko made changes -
          Link This issue relates to HDFS-601 [ HDFS-601 ]
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )
          Hide
          Konstantin Boudnik added a comment -

          _08 test case is updated according to the specs

          Show
          Konstantin Boudnik added a comment - _08 test case is updated according to the specs
          Konstantin Boudnik made changes -
          Attachment BlockReportTestPlan.html [ 12419674 ]
          Hide
          Konstantin Boudnik added a comment -

          Last three test cases were removed for they are essentially re-enforce the behavior of BlockReport_09

          Show
          Konstantin Boudnik added a comment - Last three test cases were removed for they are essentially re-enforce the behavior of BlockReport_09
          Konstantin Boudnik made changes -
          Attachment BlockReportTestPlan.html [ 12419704 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          15d 6h 51m 1 Konstantin Boudnik 03/Sep/09 03:03
          Patch Available Patch Available Resolved Resolved
          1d 14h 59m 1 Hairong Kuang 04/Sep/09 18:02
          Resolved Resolved Closed Closed
          354d 3h 46m 1 Tom White 24/Aug/10 21:49

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development