Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1789

Refactor frequently used codes from DFSOutputStream, BlockReceiver and DataXceiver

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: datanode, hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Below are frequently used codes

      • Check block tokens in DataXceiver
      • Log/error messages in BlockReceiver

      In addition, I will refactor DFSOutputStream for testing)

      • create socket for pipeline
      1. h1789_20110325.patch
        37 kB
        Tsz Wo Nicholas Sze
      2. h1789_20110325b.patch
        25 kB
        Tsz Wo Nicholas Sze
      3. h1789_20110328.patch
        10 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1789_20110325.patch: 1st patch.

          Show
          Tsz Wo Nicholas Sze added a comment - h1789_20110325.patch: 1st patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch also fixes some misleading comments/variable names. E.g.

          • Case 1
            -    private int numTargets;     // number of downstream datanodes including myself
            

            numTargets actually excludes the current datanode

          • Case 2
            -    long lastByteInBlock;
            ...
            +    final long lastByteInPacket;
            
          Show
          Tsz Wo Nicholas Sze added a comment - The patch also fixes some misleading comments/variable names. E.g. Case 1 - private int numTargets; // number of downstream datanodes including myself numTargets actually excludes the current datanode Case 2 - long lastByteInBlock; ... + final long lastByteInPacket;
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474675/h1789_20110325.patch
          against trunk revision 1085509.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 22 javac compiler warnings (more than the trunk's current 21 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474675/h1789_20110325.patch against trunk revision 1085509. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 22 javac compiler warnings (more than the trunk's current 21 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/289//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          All failed tests are known and unrelated the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - All failed tests are known and unrelated the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, just find that I have posted a wrong patch. The codes in h1789_20110325.patch is for HDFS-1761.

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, just find that I have posted a wrong patch. The codes in h1789_20110325.patch is for HDFS-1761 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1789_20110325b.patch: this is the refactoring patch.

          Show
          Tsz Wo Nicholas Sze added a comment - h1789_20110325b.patch: this is the refactoring patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474688/h1789_20110325b.patch
          against trunk revision 1085509.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474688/h1789_20110325b.patch against trunk revision 1085509. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/291//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The failed tests are not related to this and the patch is the right one.

          Show
          Tsz Wo Nicholas Sze added a comment - The failed tests are not related to this and the patch is the right one.
          Hide
          Jitendra Nath Pandey added a comment -

          1. Method PacketResponder#isLastDatanode()
          If downstreams is null for the last datanode it will return false.

          2. In PacketResponder#run
          > if (downstreams != null && downstreams.length > 0
          It can be replaced by if (!isLastDatanode())

          Show
          Jitendra Nath Pandey added a comment - 1. Method PacketResponder#isLastDatanode() If downstreams is null for the last datanode it will return false. 2. In PacketResponder#run > if (downstreams != null && downstreams.length > 0 It can be replaced by if (!isLastDatanode())
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > If downstreams is null for the last datanode it will return false.

          downstreams replaces numTargets in the original codes, numTargets could be -1, 0 and >0 where numTargets == -1 means there is no pipeline (the codes use it to do checksum.) In the new codes, we have downstreams == null for no pipeline and it is correct for isLastDatanode() to return false.

          > It can be replaced by if (!isLastDatanode())

          Because of the no pipeline case, we cannot.

          Show
          Tsz Wo Nicholas Sze added a comment - > If downstreams is null for the last datanode it will return false. downstreams replaces numTargets in the original codes, numTargets could be -1, 0 and >0 where numTargets == -1 means there is no pipeline (the codes use it to do checksum.) In the new codes, we have downstreams == null for no pipeline and it is correct for isLastDatanode() to return false. > It can be replaced by if (!isLastDatanode()) Because of the no pipeline case, we cannot.
          Hide
          Jitendra Nath Pandey added a comment -

          > downstreams == null for no pipeline
          I would prefer a more explicit way of distinguishing pipeline case and no pipeline case, but that can be addressed in a different jira.

          +1 for the patch.

          Show
          Jitendra Nath Pandey added a comment - > downstreams == null for no pipeline I would prefer a more explicit way of distinguishing pipeline case and no pipeline case, but that can be addressed in a different jira. +1 for the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > +1 system test framework. The patch passed system test framework compile.

          Try to commit the patch but the fault inject tests cannot be compiled. Seems that the test-patch script does not really work.

          Show
          Tsz Wo Nicholas Sze added a comment - > +1 system test framework. The patch passed system test framework compile. Try to commit the patch but the fault inject tests cannot be compiled. Seems that the test-patch script does not really work.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1789_20110328.patch: reverted the changes in BlockReceiver.

          Show
          Tsz Wo Nicholas Sze added a comment - h1789_20110328.patch: reverted the changes in BlockReceiver.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474843/h1789_20110328.patch
          against trunk revision 1086461.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSShell

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474843/h1789_20110328.patch against trunk revision 1086461. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSShell -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/298//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474843/h1789_20110328.patch
          against trunk revision 1086461.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474843/h1789_20110328.patch against trunk revision 1086461. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/299//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the new patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for the new patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Jitendra for the review.

          • The failed tests are not related.
          • The patch is for this issue.
          • Checked compilation locally.

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Jitendra for the review. The failed tests are not related. The patch is for this issue. Checked compilation locally. I have committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #582 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/582/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #582 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/582/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development