Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8326

Documentation about when checkpoints are run is out of date

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Apparently checkpointing by interval or transaction size are both supported in at least HDFS 2.3, but the documentation does not reflect this.

      1. HDFS-8326.001.patch
        4 kB
        Anu Engineer
      2. HDFS-8326.002.patch
        5 kB
        Misty Stanley-Jones
      3. HDFS-8326.003.patch
        5 kB
        Misty Stanley-Jones
      4. HDFS-8326.004.patch
        5 kB
        Misty Stanley-Jones
      5. HDFS-8326.patch
        4 kB
        Misty Stanley-Jones

        Activity

        Hide
        djp Junping Du added a comment -

        Looks like we wrongly add this to the CHANGES.txt entry of 2.7.2 for branch-2. Remove it from 2.7.2.

        Show
        djp Junping Du added a comment - Looks like we wrongly add this to the CHANGES.txt entry of 2.7.2 for branch-2. Remove it from 2.7.2.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to branch-2.8.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to branch-2.8.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks, Xiaoyu Yao.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks, Xiaoyu Yao .
        Hide
        xyao Xiaoyu Yao added a comment -

        Good catch, Masatake Iwasaki. I will cherry-pick the fix to branch-2.

        Show
        xyao Xiaoyu Yao added a comment - Good catch, Masatake Iwasaki . I will cherry-pick the fix to branch-2.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Xiaoyu Yao, though the "Fix version/s" is set to 2.8.0, the patch was committed to trunk only. Could you commit this to branch-2? Syncing HdfsDesign.md between trunk and branch-2 would make it easier to maintain.

        Show
        iwasakims Masatake Iwasaki added a comment - Xiaoyu Yao , though the "Fix version/s" is set to 2.8.0, the patch was committed to trunk only. Could you commit this to branch-2? Syncing HdfsDesign.md between trunk and branch-2 would make it easier to maintain.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7779/)
        HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7779/ ) HDFS-8326 . Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao) (xyao: rev d0e75e60fb16ffd6c95648a06ff3958722f71e4d) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        xyao Xiaoyu Yao added a comment -

        I remove the extra whitespace for check style warning and commit the patch to trunk. Thanks Misty and Anu for the contribution.

        Show
        xyao Xiaoyu Yao added a comment - I remove the extra whitespace for check style warning and commit the patch to trunk. Thanks Misty and Anu for the contribution.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 2m 57s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 site 2m 57s Site still builds.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
            6m 18s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731555/HDFS-8326.004.patch
        Optional Tests site
        git revision trunk / f523e96
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10882/artifact/patchprocess/whitespace.txt
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10882/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 57s Site still builds. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.     6m 18s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731555/HDFS-8326.004.patch Optional Tests site git revision trunk / f523e96 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10882/artifact/patchprocess/whitespace.txt Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10882/console This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        +1 for v004 pending Jenkins.

        Show
        xyao Xiaoyu Yao added a comment - +1 for v004 pending Jenkins.
        Hide
        misty Misty Stanley-Jones added a comment -

        Fixed more nits and moved it back to one paragraph

        Show
        misty Misty Stanley-Jones added a comment - Fixed more nits and moved it back to one paragraph
        Hide
        xyao Xiaoyu Yao added a comment -
        1. One extra space "Editlog. During".
        2. Also suggest keeping the namenode contents in single paragraph.
        Show
        xyao Xiaoyu Yao added a comment - One extra space "Editlog. During". Also suggest keeping the namenode contents in single paragraph.
        Hide
        misty Misty Stanley-Jones added a comment -

        Fixed the nits that came in while I was doing the other copy-edit and fixed some trailing whitespace.

        Show
        misty Misty Stanley-Jones added a comment - Fixed the nits that came in while I was doing the other copy-edit and fixed some trailing whitespace.
        Hide
        misty Misty Stanley-Jones added a comment -

        Copy-edited version

        Show
        misty Misty Stanley-Jones added a comment - Copy-edited version
        Hide
        xyao Xiaoyu Yao added a comment -

        The patch looks good. Just some NITs:

        1. There is an extra space in "read FsImage"
        2. "the changes from Editlog is applied " should be "the changes from Editlog are applied "
        3. "If each of these properties is set" can be reworded to "If both of these properties are set"
        Show
        xyao Xiaoyu Yao added a comment - The patch looks good. Just some NITs: There is an extra space in "read FsImage" "the changes from Editlog is applied " should be "the changes from Editlog are applied " "If each of these properties is set" can be reworded to "If both of these properties are set"
        Hide
        anu Anu Engineer added a comment -

        no worries, please go ahead, I am just trying to get this into the BB2015-05-RFC. queue.

        Show
        anu Anu Engineer added a comment - no worries, please go ahead, I am just trying to get this into the BB2015-05-RFC. queue.
        Hide
        misty Misty Stanley-Jones added a comment -

        Thanks Anu Engineer! Do you mind if I do just a little copy editing on your version?

        Show
        misty Misty Stanley-Jones added a comment - Thanks Anu Engineer ! Do you mind if I do just a little copy editing on your version?
        Hide
        misty Misty Stanley-Jones added a comment -

        Cool! I'd like to have a look at your patch to see your improvements so I can learn for next time.

        Show
        misty Misty Stanley-Jones added a comment - Cool! I'd like to have a look at your patch to see your improvements so I can learn for next time.
        Hide
        anu Anu Engineer added a comment -

        Making changes based on Andrew's comments as part of apache bug bash. Please make sure that credit for this work indeed goes to Misty.

        Show
        anu Anu Engineer added a comment - Making changes based on Andrew's comments as part of apache bug bash. Please make sure that credit for this work indeed goes to Misty.
        Hide
        anu Anu Engineer added a comment -

        > I believe the config key is "txns" not "txn"
        Fixed.

        > Would be good to put units for the period, which I believe is seconds.
        Added

        > While we're in this doc, let's remove the "This key metadata item..." sentence, since it really dates this doc and is unnecessary.
        Done

        >The bit about a consistency tradeoff is not quite right. We write to the edit log because it's very efficient, more efficient than trying to update the fsimage in place. We need to checkpoint to prevent an unbounded number of edits, which is bad because it fills up >disks and also delays NN startup time since the edits need to be applied to the base fsimage. You might consider taking some content from the blog post I wrote:

        I was not sure if could copy from your blog, so I re-wrote the same idea in the documentation and now it is more aligned to what you have expressed in the blog

        Show
        anu Anu Engineer added a comment - > I believe the config key is "txns" not "txn" Fixed. > Would be good to put units for the period, which I believe is seconds. Added > While we're in this doc, let's remove the "This key metadata item..." sentence, since it really dates this doc and is unnecessary. Done >The bit about a consistency tradeoff is not quite right. We write to the edit log because it's very efficient, more efficient than trying to update the fsimage in place. We need to checkpoint to prevent an unbounded number of edits, which is bad because it fills up >disks and also delays NN startup time since the edits need to be applied to the base fsimage. You might consider taking some content from the blog post I wrote: I was not sure if could copy from your blog, so I re-wrote the same idea in the documentation and now it is more aligned to what you have expressed in the blog
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Misty, a few review comments:

        • I believe the config key is "txns" not "txn"
        • Would be good to put units for the period, which I believe is seconds.
        • While we're in this doc, let's remove the "This key metadata item..." sentence, since it really dates this doc and is unnecessary.
        • The bit about a consistency tradeoff is not quite right. We write to the edit log because it's very efficient, more efficient than trying to update the fsimage in place. We need to checkpoint to prevent an unbounded number of edits, which is bad because it fills up disks and also delays NN startup time since the edits need to be applied to the base fsimage. You might consider taking some content from the blog post I wrote:

        http://blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/

        Show
        andrew.wang Andrew Wang added a comment - Hi Misty, a few review comments: I believe the config key is "txns" not "txn" Would be good to put units for the period, which I believe is seconds. While we're in this doc, let's remove the "This key metadata item..." sentence, since it really dates this doc and is unnecessary. The bit about a consistency tradeoff is not quite right. We write to the edit log because it's very efficient, more efficient than trying to update the fsimage in place. We need to checkpoint to prevent an unbounded number of edits, which is bad because it fills up disks and also delays NN startup time since the edits need to be applied to the base fsimage. You might consider taking some content from the blog post I wrote: http://blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 2m 53s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 site 2m 54s Site still builds.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
            6m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730529/HDFS-8326.patch
        Optional Tests site
        git revision trunk / 05adc76
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10813/artifact/patchprocess/whitespace.txt
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10813/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 54s Site still builds. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.     6m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730529/HDFS-8326.patch Optional Tests site git revision trunk / 05adc76 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10813/artifact/patchprocess/whitespace.txt Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10813/console This message was automatically generated.
        Hide
        misty Misty Stanley-Jones added a comment -

        Here is my attempt to update the documentation.

        Show
        misty Misty Stanley-Jones added a comment - Here is my attempt to update the documentation.

          People

          • Assignee:
            misty Misty Stanley-Jones
            Reporter:
            misty Misty Stanley-Jones
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development