Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-599

Improve Namenode robustness by prioritizing datanode heartbeats over client requests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      fb

      Description

      The namenode processes RPC requests from clients that are reading/writing to files as well as heartbeats/block reports from datanodes.

      Sometime, because of various reasons (Java GC runs, inconsistent performance of NFS filer that stores HDFS transacttion logs, etc), the namenode encounters transient slowness. For example, if the device that stores the HDFS transaction logs becomes sluggish, the Namenode's ability to process RPCs slows down to a certain extent. During this time, the RPCs from clients as well as the RPCs from datanodes suffer in similar fashion. If the underlying problem becomes worse, the NN's ability to process a heartbeat from a DN is severly impacted, thus causing the NN to declare that the DN is dead. Then the NN starts replicating blocks that used to reside on the now-declared-dead datanode. This adds extra load to the NN. Then the now-declared-datanode finally re-establishes contact with the NN, and sends a block report. The block report processing on the NN is another heavyweight activity, thus casing more load to the already overloaded namenode.

      My proposal is tha the NN should try its best to continue processing RPCs from datanodes and give lesser priority to serving client requests. The Datanode RPCs are integral to the consistency and performance of the Hadoop file system, and it is better to protect it at all costs. This will ensure that NN recovers from the hiccup much faster than what it does now.

      1. HDFS-599.y20.patch
        17 kB
        Suresh Srinivas
      2. HDFS-599.patch
        23 kB
        Dmytro Molkov
      3. HDFS-599.3.patch
        20 kB
        Dmytro Molkov

        Issue Links

          Activity

          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-1392 [ HDFS-1392 ]
          Kan Zhang made changes -
          Link This issue is related to HDFS-1357 [ HDFS-1357 ]
          gary murry made changes -
          Link This issue relates to HDFS-1321 [ HDFS-1321 ]
          Suresh Srinivas made changes -
          Attachment HDFS-599.y20.patch [ 12449119 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-1291 [ HDFS-1291 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HBASE-2782 [ HBASE-2782 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314241 ]
          Dmytro Molkov made changes -
          Attachment HDFS-599.3.patch [ 12445580 ]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Dmytro Molkov made changes -
          Link This issue blocks HDFS-1156 [ HDFS-1156 ]
          Dmytro Molkov made changes -
          Link This issue is related to HADOOP-6764 [ HADOOP-6764 ]
          Dmytro Molkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Dmytro Molkov made changes -
          Attachment HDFS-599.patch [ 12443022 ]
          Dmytro Molkov made changes -
          Link This issue is blocked by HADOOP-6469 [ HADOOP-6469 ]
          Dmytro Molkov made changes -
          Assignee dhruba borthakur [ dhruba ] Dmytro Molkov [ dms ]
          dhruba borthakur made changes -
          Field Original Value New Value
          Tags fb
          dhruba borthakur created issue -

            People

            • Assignee:
              Dmytro Molkov
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development