Hadoop Common
  1. Hadoop Common
  2. HADOOP-5034

NameNode should send both replication and deletion requests to DataNode in one reply to a heartbeat

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch changes the DatanodeProtocoal version number from 18 to 19. The patch allows NameNode to send both block replication and deletion request to a DataNode in response to a heartbeat.

      Description

      Currently NameNode favors block replication requests over deletion requests. On reply to a heartbeat, NameNode does not send a block deletion request unless there is no block replication request.

      This brings a problem when a near-full cluster loses a bunch of DataNodes. In react to the DataNode loss, NameNode starts to replicate blocks. However, replication takes a lot of cpu and a lot of replications fail because of the lack of disk space. So the administrator tries to delete some DFS files to free up space. However, block deletion requests get delayed for very long time because it takes a long time to drain the block replication requests for most DataNodes.

      I'd like to propose to let NameNode to send both replication requests and deletion requests to DataNodes in one reply to a heartbeat. This also implies that the replication monitor should schedule both replication and deletion work in one iteration.

      1. blockTransferInvalidate3.patch
        14 kB
        Hairong Kuang
      2. blockTransferInvalidate2.patch
        9 kB
        Hairong Kuang
      3. blockTransferInvalidate1.patch
        14 kB
        Hairong Kuang
      4. blockTransferInvalidate.patch
        12 kB
        Hairong Kuang

        Activity

        Hide
        Robert Chansler added a comment -

        I first thought that just switching the priority to deletions rather than replications would be satisfactory, but Hairong explained that since processing is time-sliced, if the higher priority task occurs at the very modest rate of once per heartbeat, the lower priority task will be starved. Starving deletions can make replication impossible. The reverse is not true, but it is difficult to make the statistical argument that starving replications for a while is OK. You can't starve replication too long by doing deletions as there are only so many replicas to delete, but is it OK for replications to wait a minute? An hour? It is perhaps best to just follow Hairong's suggestion.

        Show
        Robert Chansler added a comment - I first thought that just switching the priority to deletions rather than replications would be satisfactory, but Hairong explained that since processing is time-sliced, if the higher priority task occurs at the very modest rate of once per heartbeat, the lower priority task will be starved. Starving deletions can make replication impossible. The reverse is not true, but it is difficult to make the statistical argument that starving replications for a while is OK. You can't starve replication too long by doing deletions as there are only so many replicas to delete, but is it OK for replications to wait a minute? An hour? It is perhaps best to just follow Hairong's suggestion.
        Hide
        Konstantin Shvachko added a comment -

        I think we should first try to fine-tune the number of blocks we send for deletion. Currently we send 100, which means that data-node can not delete more than a 100 blocks every 3 seconds. This seems to be low on large clusters, but I don't what the optimum is.
        If fine-tuning will not solve the problem we can go on with the modifications to the code.

        Show
        Konstantin Shvachko added a comment - I think we should first try to fine-tune the number of blocks we send for deletion. Currently we send 100, which means that data-node can not delete more than a 100 blocks every 3 seconds. This seems to be low on large clusters, but I don't what the optimum is. If fine-tuning will not solve the problem we can go on with the modifications to the code.
        Hide
        Raghu Angadi added a comment -

        > I think we should first try to fine-tune the number of blocks we send for deletion. Currently we send 100 [...]

        I think this limit is related but different issue. Even when it was implemented it was supposed to be a work around for how DN handles deletion. We should either remove or have a very large limit at NN and let DN handle deleting large number of blocks properly (say in a separate thread from heart beat thread). This was fix was proposed quite a few times but we didn't fix it. Trying to fine tune it only prolongs the problem.

        > If fine-tuning will not solve the problem we can go on with the modifications to the code.

        It does not look like changing this limit won't fix the issue since NN never get to send any block to delete. Logically I don't see any reason why NN can not send both replication and deletion requests in the same response to DN.

        Show
        Raghu Angadi added a comment - > I think we should first try to fine-tune the number of blocks we send for deletion. Currently we send 100 [...] I think this limit is related but different issue. Even when it was implemented it was supposed to be a work around for how DN handles deletion. We should either remove or have a very large limit at NN and let DN handle deleting large number of blocks properly (say in a separate thread from heart beat thread). This was fix was proposed quite a few times but we didn't fix it. Trying to fine tune it only prolongs the problem. > If fine-tuning will not solve the problem we can go on with the modifications to the code. It does not look like changing this limit won't fix the issue since NN never get to send any block to delete. Logically I don't see any reason why NN can not send both replication and deletion requests in the same response to DN.
        Hide
        Hairong Kuang added a comment -

        I do not think fine-tuning the limit will solve the problem. What happened was that block replication starved block deletion. Block deletion was not observed at all. When I ReplicationMonitor code, I found out that block deletion does not get scheduled for any datanode even if there is only one replication work scheduled for the whole cluster. This explains why no block deletion was observed at all.

        Show
        Hairong Kuang added a comment - I do not think fine-tuning the limit will solve the problem. What happened was that block replication starved block deletion. Block deletion was not observed at all. When I ReplicationMonitor code, I found out that block deletion does not get scheduled for any datanode even if there is only one replication work scheduled for the whole cluster. This explains why no block deletion was observed at all.
        Hide
        Hairong Kuang added a comment -

        Here is the patch that does the following:
        1. ReplicationMonitor schedules both block replication and deletion in one iteration;
        2. Heartbeat picks up both replication and deletion requests if both are scheduled;
        3. BlockCommand is enhanced so that a command can carry both replication and deletion requests;
        4. Datanode handles both replication and deletion requests when receive a heartbeat reply.

        Show
        Hairong Kuang added a comment - Here is the patch that does the following: 1. ReplicationMonitor schedules both block replication and deletion in one iteration; 2. Heartbeat picks up both replication and deletion requests if both are scheduled; 3. BlockCommand is enhanced so that a command can carry both replication and deletion requests; 4. Datanode handles both replication and deletion requests when receive a heartbeat reply.
        Hide
        Raghu Angadi added a comment -

        +1 for the solution and I think this should go into 0.19 and up.. if not 0.18. Main reason is that when we are affected by this issue, there is no work around. It is pretty much a dead-lock : e.g. : if the cluster is almost ful, the replications don't succeed, but if admin wants make room by deleting file, blocks won't be deleted since there are replication pending.

        Regd implementation, the patch adds new datanode heartbeat command that combines invalidates and replications. I think it will simpler if we make heartbeat reply to contain list or array of commands. This way we don't need new command types.

        Show
        Raghu Angadi added a comment - +1 for the solution and I think this should go into 0.19 and up.. if not 0.18. Main reason is that when we are affected by this issue, there is no work around. It is pretty much a dead-lock : e.g. : if the cluster is almost ful, the replications don't succeed, but if admin wants make room by deleting file, blocks won't be deleted since there are replication pending. Regd implementation, the patch adds new datanode heartbeat command that combines invalidates and replications. I think it will simpler if we make heartbeat reply to contain list or array of commands. This way we don't need new command types.
        Hide
        dhruba borthakur added a comment -

        +1 for putting this into 0.19. Very helpful patch.

        Show
        dhruba borthakur added a comment - +1 for putting this into 0.19. Very helpful patch.
        Hide
        Hairong Kuang added a comment -

        > I think it will simpler if we make heartbeat reply to contain list or array of commands. This way we don't need new command types.
        I like this idea a lot. This makes the solution simpler and make the protocol easy to extend later on.

        > putting this into 0.19.
        Not a problem for me. This patch has a protocol change. If the community wants it in 0.19, maybe we can make an exception.

        Show
        Hairong Kuang added a comment - > I think it will simpler if we make heartbeat reply to contain list or array of commands. This way we don't need new command types. I like this idea a lot. This makes the solution simpler and make the protocol easy to extend later on. > putting this into 0.19. Not a problem for me. This patch has a protocol change. If the community wants it in 0.19, maybe we can make an exception.
        Hide
        Raghu Angadi added a comment -

        +1 for 0.19.

        Show
        Raghu Angadi added a comment - +1 for 0.19.
        Hide
        Hairong Kuang added a comment -

        This patch reflects Raghu's suggestion.

        Show
        Hairong Kuang added a comment - This patch reflects Raghu's suggestion.
        Hide
        Hairong Kuang added a comment -

        The previous patch does not include the test case.

        Show
        Hairong Kuang added a comment - The previous patch does not include the test case.
        Hide
        Raghu Angadi added a comment -

        +1 for the patch. Could we change the fix version to 0.19?

        Show
        Raghu Angadi added a comment - +1 for the patch. Could we change the fix version to 0.19?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12398963/blockTransferInvalidate3.patch
        against trunk revision 738944.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12398963/blockTransferInvalidate3.patch against trunk revision 738944. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3776/console This message is automatically generated.
        Hide
        Hairong Kuang added a comment -

        I've just committed this.

        Show
        Hairong Kuang added a comment - I've just committed this.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #756 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/ )

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development