Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1520

HDFS 20 append: Lightweight NameNode operation to trigger lease recovery

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.20-append, 0.20.205.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently HBase uses append to trigger the close of HLog during Hlog split. Append is a very expensive operation, which involves not only NameNode operations but creating a writing pipeline. If one of datanodes on the pipeline has a problem, this recovery may takes minutes. I'd like implement a lightweight NameNode operation to trigger lease recovery and make HBase to use this instead.

      1. recoverLeaseApache20.patch
        19 kB
        Hairong Kuang
      2. HDFS-1520.20-security.1.patch
        20 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          This should go in trunk as well right?

          Show
          Eli Collins added a comment - This should go in trunk as well right?
          Hide
          Hairong Kuang added a comment -

          I do not think this needs to go into trunk for now. With trunk, HBase should be able to read the unclosed file and get all hflushed data. So there is no need to close HLog before splitting.

          Show
          Hairong Kuang added a comment - I do not think this needs to go into trunk for now. With trunk, HBase should be able to read the unclosed file and get all hflushed data. So there is no need to close HLog before splitting.
          Hide
          Jonathan Gray added a comment -

          In HBase, we may want to be able to prevent the existing writer from doing any further writing. Not sure how exactly this relates to that but if we had that option on log splitting, we'd use it as we do not expect writes during the log split.

          On the replay, we might want to actually tail the split up logs while the master splits them (not now but maybe in the future), so we wouldn't want to prevent concurrent writing in that case.

          Show
          Jonathan Gray added a comment - In HBase, we may want to be able to prevent the existing writer from doing any further writing. Not sure how exactly this relates to that but if we had that option on log splitting, we'd use it as we do not expect writes during the log split. On the replay, we might want to actually tail the split up logs while the master splits them (not now but maybe in the future), so we wouldn't want to prevent concurrent writing in that case.
          Hide
          Hairong Kuang added a comment -

          Here comes the patch for hadoop append branch.

          Jonathan, if hbase needs this for HDFS trunk, I will make one for trunk too.

          Show
          Hairong Kuang added a comment - Here comes the patch for hadoop append branch. Jonathan, if hbase needs this for HDFS trunk, I will make one for trunk too.
          Hide
          Jonathan Gray added a comment -

          @Hairong, we have not worked on HBase atop 0.22 so don't worry about it now. Let's have a more in depth conversation about this when we actually support 0.22 (unless it's now or never onto trunk? in that case if it's not much trouble I'd say we want it on trunk).

          Show
          Jonathan Gray added a comment - @Hairong, we have not worked on HBase atop 0.22 so don't worry about it now. Let's have a more in depth conversation about this when we actually support 0.22 (unless it's now or never onto trunk? in that case if it's not much trouble I'd say we want it on trunk).
          Hide
          dhruba borthakur added a comment -

          +1 code looks good. I also ran all the new unit tests.

          Show
          dhruba borthakur added a comment - +1 code looks good. I also ran all the new unit tests.
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Hairong!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Hairong!
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #1697 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1697/)

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #1697 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1697/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/ )
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for 20-security.

          Show
          Jitendra Nath Pandey added a comment - Patch for 20-security.
          Hide
          Suresh Srinivas added a comment -

          +1 for the 0.20-security patch.

          Show
          Suresh Srinivas added a comment - +1 for the 0.20-security patch.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to 0.20-security branch.

          Show
          Suresh Srinivas added a comment - I committed the patch to 0.20-security branch.
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development