Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-793

DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.2, 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      Currently BlockReceiver#PacketResponder interleaves receiving ack message and sending ack message for the same packet. It reads a portion of the message, sends a portion of its ack, and continues like this until it hits the end of the message. The problem is that if it gets an error receiving the ack, it is not able to send an ack that reflects the source of the error.

      The PacketResponder should receives the whole packet ack message first and then constuct and sends out its ack.

      1. separateSendRcvAck-0.20-yahoo.patch
        14 kB
        Hairong Kuang
      2. separateSendRcvAck2.patch
        18 kB
        Hairong Kuang
      3. separateSendRcvAck-0.20.patch
        14 kB
        Hairong Kuang
      4. separateSendRcvAck1.patch
        18 kB
        Hairong Kuang
      5. separateSendRcvAck.patch
        16 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          Here is a patch for review.

          Thank Cos for his help with fixing the broken aspect bindings caused by the change made in this patch.

          Show
          Hairong Kuang added a comment - Here is a patch for review. Thank Cos for his help with fixing the broken aspect bindings caused by the change made in this patch.
          Hide
          Hairong Kuang added a comment -

          This patch has the following chanages:
          1. Bump up the data transfer protocol version number;
          2. Remove the unused method ackReply;
          3. Replace the use of seqno of -1 with PipelineAck.HEART_BEAT.getseqno();

          Show
          Hairong Kuang added a comment - This patch has the following chanages: 1. Bump up the data transfer protocol version number; 2. Remove the unused method ackReply; 3. Replace the use of seqno of -1 with PipelineAck.HEART_BEAT.getseqno();
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427277/separateSendRcvAck1.patch
          against trunk revision 888218.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427277/separateSendRcvAck1.patch against trunk revision 888218. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/135/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #135 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/135/)
          . Data node should receive the whole packet ack message before it constructs and sends its own ack message for the packet. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #135 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/135/ ) . Data node should receive the whole packet ack message before it constructs and sends its own ack message for the packet. Contributed by Hairong Kuang.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/136/)
          . Set the data transfer protocol to be 19.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/136/ ) . Set the data transfer protocol to be 19.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #137 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/137/ )
          Hide
          Hairong Kuang added a comment -

          a patch for 0.20.

          Show
          Hairong Kuang added a comment - a patch for 0.20.
          Hide
          Hairong Kuang added a comment -

          a patch for the trunk and 0.21.

          Show
          Hairong Kuang added a comment - a patch for the trunk and 0.21.
          Hide
          Hairong Kuang added a comment -

          I've just committed this.

          Show
          Hairong Kuang added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #83 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/83/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #170 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/170/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #172 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/172/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #172 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/172/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/202/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/202/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #99 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/99/)
          Move the change logs of and HDFS-101 from 0.20 section to 0.21 section

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #99 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/99/ ) Move the change logs of and HDFS-101 from 0.20 section to 0.21 section
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/ )
          Hide
          Hairong Kuang added a comment -

          This patch is for yahoo 0.20 branch.

          Show
          Hairong Kuang added a comment - This patch is for yahoo 0.20 branch.
          Hide
          Nicolas Spiegelberg added a comment -

          This should be pulled into the branch-0.20-append branch.

          Show
          Nicolas Spiegelberg added a comment - This should be pulled into the branch-0.20-append branch.
          Hide
          Todd Lipcon added a comment -

          Removing 0.20-append, since it was already applied to 20 branch in the form of HDFS-872.

          Show
          Todd Lipcon added a comment - Removing 0.20-append, since it was already applied to 20 branch in the form of HDFS-872 .

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development