Hadoop Common
  1. Hadoop Common
  2. HADOOP-5479

NameNode should not send empty block replication request to DataNode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      On our production clusters, we occasionally see that NameNode sends an empty block replication request to DataNode on every heartbeat, thus blocking this DataNode from replicating or deleting any block.

      This is partly caused by DataNode sending a wrong number of replications in progress which will be fixed by HADOOP-5465. There is also a flaw at the NameNode side. NameNode should not interpret the number of replications in progress as the number of targets since replication is done through a pipeline. It also should make sure that no empty replication request is sent to DataNode.

      1. numTransfers.patch
        3 kB
        Hairong Kuang
      2. numTransfers1.patch
        3 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          This patch has three changes:

          1. NameNode interprets numOfTransfers as numOfBlocks to be replicated. The current code interprets it as numOfTargets to be replicated. This change is made in DatanodeDescriptor#BlockTargetPair.poll(). This prevents empty replication requests as well as empty recover requests.
          2. The number of targets to be chosen is not capped by the number of transfers. Again NameNode should not treat the number of transfers as the number of targets.
          3. The third change is not directly related to this issue. But I saw this happen when I debugged this issue. The current code moves a block to the pending replication queue only when it reaches its replication factor. This sometimes causes over-replication because it does not track all pending replications. This patch adds a block to the pending replication queue whenever there is one replication scheduled for this block.
          Show
          Hairong Kuang added a comment - This patch has three changes: NameNode interprets numOfTransfers as numOfBlocks to be replicated. The current code interprets it as numOfTargets to be replicated. This change is made in DatanodeDescriptor#BlockTargetPair.poll(). This prevents empty replication requests as well as empty recover requests. The number of targets to be chosen is not capped by the number of transfers. Again NameNode should not treat the number of transfers as the number of targets. The third change is not directly related to this issue. But I saw this happen when I debugged this issue. The current code moves a block to the pending replication queue only when it reaches its replication factor. This sometimes causes over-replication because it does not track all pending replications. This patch adds a block to the pending replication queue whenever there is one replication scheduled for this block.
          Hide
          Konstantin Shvachko added a comment -

          +1 on all changes.

          Little thing:
          else clause in DatanodeDescriptor.BlockQueue.poll() can be removed.

          Show
          Konstantin Shvachko added a comment - +1 on all changes. Little thing: else clause in DatanodeDescriptor.BlockQueue.poll() can be removed.
          Hide
          Hairong Kuang added a comment -

          Thanks Konstantin for the review. This patch fixed the little thing as well.

          Show
          Hairong Kuang added a comment - Thanks Konstantin for the review. This patch fixed the little thing as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12402322/numTransfers1.patch
          against trunk revision 755003.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12402322/numTransfers1.patch against trunk revision 755003. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/92/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          I've just committed this.

          Show
          Hairong Kuang added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #783 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/783/)
          . NameNode should not send empty block replication request to DataNode. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #783 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/783/ ) . NameNode should not send empty block replication request to DataNode. Contributed by Hairong Kuang.
          Hide
          Thanh Do added a comment -

          Hi Hairong,

          This is an old issue but could you please verify my understanding about this bug? So basically, because the NN keeps send empty replication command, the DN creates a new thread for every empty command, which eats up DN's resource, right?

          Many thanks

          Show
          Thanh Do added a comment - Hi Hairong, This is an old issue but could you please verify my understanding about this bug? So basically, because the NN keeps send empty replication command, the DN creates a new thread for every empty command, which eats up DN's resource, right? Many thanks

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development