Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4721

Speed up lease/block recovery when DN fails and a block goes into recovery

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This was observed while doing HBase WAL recovery. HBase uses append to write to its write ahead log. So initially the pipeline is setup as

      DN1 --> DN2 --> DN3

      This WAL needs to be read when DN1 fails since it houses the HBase regionserver for the WAL.

      HBase first recovers the lease on the WAL file. During recovery, we choose DN1 as the primary DN to do the recovery even though DN1 has failed and is not heartbeating any more.

      Avoiding the stale DN1 would speed up recovery and reduce hbase MTTR. There are two options.
      a) Ride on HDFS 3703 and if stale node detection is turned on, we do not choose stale datanodes (typically not heart beated for 20-30 seconds) as primary DN(s)
      b) We sort the replicas in order of last heart beat and always pick the ones which gave the most recent heart beat

      Going to the dead datanode increases lease + block recovery since the block goes into UNDER_RECOVERY state even though no one is recovering it actively. Please let me know if this makes sense. If yes, whether we should move forward with a) or b).

      Thanks

      1. 4721-trunk-v4.patch
        23 kB
        Varun Sharma
      2. 4721-branch2.patch
        23 kB
        Varun Sharma
      3. 4721-trunk-v3.patch
        22 kB
        Varun Sharma
      4. 4721-trunk-v2.patch
        20 kB
        Varun Sharma
      5. 4721-trunk.patch
        20 kB
        Varun Sharma
      6. 4721-v8.patch
        21 kB
        Varun Sharma
      7. 4721-v7.patch
        21 kB
        Varun Sharma
      8. 4721-v6.patch
        21 kB
        Varun Sharma
      9. 4721-v5.patch
        18 kB
        Varun Sharma
      10. 4721-v4.patch
        6 kB
        Varun Sharma
      11. 4721-v3.patch
        6 kB
        Varun Sharma
      12. 4721-v2.patch
        11 kB
        Varun Sharma

        Issue Links

          Activity

          Varun Sharma created issue -
          Varun Sharma made changes -
          Field Original Value New Value
          Attachment 4721-hadoop2.patch [ 12579764 ]
          Varun Sharma made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 2.0.4-alpha [ 12324136 ]
          Nicolas Liochon made changes -
          Link This issue is required by HBASE-5843 [ HBASE-5843 ]
          Ted Yu made changes -
          Link This issue relates to HDFS-4724 [ HDFS-4724 ]
          Varun Sharma made changes -
          Attachment 4721-v2.patch [ 12579904 ]
          Varun Sharma made changes -
          Attachment 4721-v3.patch [ 12579932 ]
          Varun Sharma made changes -
          Attachment 4721-v4.patch [ 12579964 ]
          Varun Sharma made changes -
          Attachment 4721-v4.patch [ 12579964 ]
          Varun Sharma made changes -
          Attachment 4721-v4.patch [ 12579965 ]
          Varun Sharma made changes -
          Attachment 4721-v5.patch [ 12580008 ]
          Varun Sharma made changes -
          Attachment 4721-v6.patch [ 12580187 ]
          Varun Sharma made changes -
          Attachment 4721-v6.patch [ 12580187 ]
          Varun Sharma made changes -
          Attachment 4721-v6.patch [ 12580190 ]
          Varun Sharma made changes -
          Attachment 4721-v7.patch [ 12580200 ]
          Varun Sharma made changes -
          Attachment 4721-v8.patch [ 12580201 ]
          Varun Sharma made changes -
          Attachment 4721-trunk.patch [ 12580215 ]
          Varun Sharma made changes -
          Attachment 4721-trunk-v2.patch [ 12580300 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HBASE-8389 [ HBASE-8389 ]
          Varun Sharma made changes -
          Attachment 4721-trunk-v3.patch [ 12580494 ]
          Nicolas Liochon made changes -
          Link This issue relates to HDFS-4754 [ HDFS-4754 ]
          Varun Sharma made changes -
          Attachment 4721-hadoop2.patch [ 12579764 ]
          Varun Sharma made changes -
          Attachment 4721-branch2.patch [ 12580721 ]
          Varun Sharma made changes -
          Attachment 4721-trunk-v4.patch [ 12580722 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Assignee Varun Sharma [ varunsharma ]
          Fix Version/s 2.0.4-alpha [ 12324136 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.0.5-beta [ 12324031 ]
          Resolution Fixed [ 1 ]
          Ted Yu made changes -
          Link This issue is related to HDFS-4796 [ HDFS-4796 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Varun Sharma
              Reporter:
              Varun Sharma
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development