Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-793

DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.2, 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      Currently BlockReceiver#PacketResponder interleaves receiving ack message and sending ack message for the same packet. It reads a portion of the message, sends a portion of its ack, and continues like this until it hits the end of the message. The problem is that if it gets an error receiving the ack, it is not able to send an ack that reflects the source of the error.

      The PacketResponder should receives the whole packet ack message first and then constuct and sends out its ack.

      1. separateSendRcvAck2.patch
        18 kB
        Hairong Kuang
      2. separateSendRcvAck1.patch
        18 kB
        Hairong Kuang
      3. separateSendRcvAck-0.20-yahoo.patch
        14 kB
        Hairong Kuang
      4. separateSendRcvAck-0.20.patch
        14 kB
        Hairong Kuang
      5. separateSendRcvAck.patch
        16 kB
        Hairong Kuang

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is depended upon by HDFS-142 [ HDFS-142 ]
          Gavin made changes -
          Link This issue blocks HDFS-142 [ HDFS-142 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Todd Lipcon made changes -
          Affects Version/s 0.20-append [ 12315103 ]
          Hide
          Todd Lipcon added a comment -

          Removing 0.20-append, since it was already applied to 20 branch in the form of HDFS-872.

          Show
          Todd Lipcon added a comment - Removing 0.20-append, since it was already applied to 20 branch in the form of HDFS-872 .
          Nicolas Spiegelberg made changes -
          Link This issue blocks HDFS-142 [ HDFS-142 ]
          Nicolas Spiegelberg made changes -
          Affects Version/s 0.20-append [ 12315103 ]
          Hide
          Nicolas Spiegelberg added a comment -

          This should be pulled into the branch-0.20-append branch.

          Show
          Nicolas Spiegelberg added a comment - This should be pulled into the branch-0.20-append branch.
          Tom White made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Hairong Kuang made changes -
          Attachment separateSendRcvAck-0.20-yahoo.patch [ 12437808 ]
          Hide
          Hairong Kuang added a comment -

          This patch is for yahoo 0.20 branch.

          Show
          Hairong Kuang added a comment - This patch is for yahoo 0.20 branch.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #99 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/99/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #99 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/99/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/202/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/202/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #172 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/172/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #172 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/172/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Allen Wittenauer made changes -
          Link This issue relates to HDFS-872 [ HDFS-872 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/ )
          Hide
          Hairong Kuang added a comment -

          I've just committed this.

          Show
          Hairong Kuang added a comment - I've just committed this.
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hairong Kuang made changes -
          Attachment separateSendRcvAck2.patch [ 12427428 ]
          Hide
          Hairong Kuang added a comment -

          a patch for the trunk and 0.21.

          Show
          Hairong Kuang added a comment - a patch for the trunk and 0.21.
          Hairong Kuang made changes -
          Attachment separateSendRcvAck-0.20.patch [ 12427427 ]
          Hide
          Hairong Kuang added a comment -

          a patch for 0.20.

          Show
          Hairong Kuang added a comment - a patch for 0.20.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/136/)
          . Set the data transfer protocol to be 19.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/136/ ) . Set the data transfer protocol to be 19.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #135 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/135/)
          . Data node should receive the whole packet ack message before it constructs and sends its own ack message for the packet. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #135 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/135/ ) . Data node should receive the whole packet ack message before it constructs and sends its own ack message for the packet. Contributed by Hairong Kuang.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Incompatible change, Reviewed]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427277/separateSendRcvAck1.patch
          against trunk revision 888218.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427277/separateSendRcvAck1.patch against trunk revision 888218. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/console This message is automatically generated.
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Attachment separateSendRcvAck1.patch [ 12427277 ]
          Hairong Kuang made changes -
          Attachment separateSendRcvAck1.patch [ 12427275 ]
          Hairong Kuang made changes -
          Attachment separateSendRcvAck1.patch [ 12427275 ]
          Hide
          Hairong Kuang added a comment -

          This patch has the following chanages:
          1. Bump up the data transfer protocol version number;
          2. Remove the unused method ackReply;
          3. Replace the use of seqno of -1 with PipelineAck.HEART_BEAT.getseqno();

          Show
          Hairong Kuang added a comment - This patch has the following chanages: 1. Bump up the data transfer protocol version number; 2. Remove the unused method ackReply; 3. Replace the use of seqno of -1 with PipelineAck.HEART_BEAT.getseqno();
          Hairong Kuang made changes -
          Attachment separateSendRcvAck.patch [ 12427009 ]
          Hide
          Hairong Kuang added a comment -

          Here is a patch for review.

          Thank Cos for his help with fixing the broken aspect bindings caused by the change made in this patch.

          Show
          Hairong Kuang added a comment - Here is a patch for review. Thank Cos for his help with fixing the broken aspect bindings caused by the change made in this patch.
          Hairong Kuang made changes -
          Field Original Value New Value
          Link This issue blocks HDFS-101 [ HDFS-101 ]
          Hairong Kuang created issue -

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development