Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1596

Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removed references to the older fs.checkpoint.* properties that resided in core-site.xml
    • Tags:
      checkpoint, secondarynamenode

      Description

      The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.

      <property>
      <name>fs.checkpoint.dir</name>
      <value>$

      {hadoop.tmp.dir}

      /dfs/namesecondary</value>
      <description>Determines where on the local filesystem the DFS secondary
      name node should store the temporary images to merge.
      If this is a comma-delimited list of directories then the image is
      replicated in all of the directories for redundancy.
      </description>
      </property>

      <property>
      <name>fs.checkpoint.edits.dir</name>
      <value>$

      {fs.checkpoint.dir}

      </value>
      <description>Determines where on the local filesystem the DFS secondary
      name node should store the temporary edits to merge.
      If this is a comma-delimited list of directoires then teh edits is
      replicated in all of the directoires for redundancy.
      Default value is same as fs.checkpoint.dir
      </description>
      </property>

      <property>
      <name>fs.checkpoint.period</name>
      <value>3600</value>
      <description>The number of seconds between two periodic checkpoints.
      </description>
      </property>

      <property>
      <name>fs.checkpoint.size</name>
      <value>67108864</value>
      <description>The size of the current edit log (in bytes) that triggers
      a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
      </description>
      </property>

      1. HDFS-7117.r1.diff
        6 kB
        Harsh J
      2. HDFS-7117.r2.diff
        4 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Haven't these keys been deprecated, in favor of "dfs.namenode.checkpoint.*" prefixes, available in hdfs-default.xml? Is the move still required? Just clarifying before I attempt to do this.

          Show
          Harsh J added a comment - Haven't these keys been deprecated, in favor of "dfs.namenode.checkpoint.*" prefixes, available in hdfs-default.xml? Is the move still required? Just clarifying before I attempt to do this.
          Hide
          Patrick Angeles added a comment -

          I still see them in core-default.xml in trunk, so they should get removed or the deprecation should be documented in the xml file itself...

          Show
          Patrick Angeles added a comment - I still see them in core-default.xml in trunk, so they should get removed or the deprecation should be documented in the xml file itself...
          Hide
          Harsh J added a comment -

          Patch that updates all references of fs.checkpoint.* to their newer dfs.namenode.checkpoint.* keys.

          Show
          Harsh J added a comment - Patch that updates all references of fs.checkpoint.* to their newer dfs.namenode.checkpoint.* keys.
          Hide
          Harsh J added a comment -

          Marking as Patch-Available.

          Show
          Harsh J added a comment - Marking as Patch-Available.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12469584/HDFS-7117.r1.diff
          against trunk revision 1075571.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/217//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469584/HDFS-7117.r1.diff against trunk revision 1075571. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/217//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Ouch, looks like my IDE's re-arrangement of imports had struck again (Wasn't disabled then, is always disabled now).

          Cleaned up the patch and re-upped. It can be applied cleanly now.

          Show
          Harsh J added a comment - Ouch, looks like my IDE's re-arrangement of imports had struck again (Wasn't disabled then, is always disabled now). Cleaned up the patch and re-upped. It can be applied cleanly now.
          Hide
          Harsh J added a comment -

          Re-marking as PA.

          Show
          Harsh J added a comment - Re-marking as PA.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12473714/HDFS-7117.r2.diff
          against trunk revision 1081580.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473714/HDFS-7117.r2.diff against trunk revision 1081580. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/261//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          I have not a clue why these test cases started to fail with this patch. They do not seem related to the removal of these property refs (except from the area that marks it as deprecated).

          Is it due to the HADOOP-7117 co-requirement [which involved changes to core-site.xml]?

          Show
          Harsh J added a comment - I have not a clue why these test cases started to fail with this patch. They do not seem related to the removal of these property refs (except from the area that marks it as deprecated). Is it due to the HADOOP-7117 co-requirement [which involved changes to core-site.xml] ?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I have not a clue why these test cases started to fail with this patch. ...

          This is now just documentation changes. The failed tests are not related.

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - > I have not a clue why these test cases started to fail with this patch. ... This is now just documentation changes. The failed tests are not related. +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Harsh!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Harsh!
          Hide
          Harsh J added a comment -

          Great!

          Could you also have a look at HADOOP-7117? Thank you!

          Show
          Harsh J added a comment - Great! Could you also have a look at HADOOP-7117 ? Thank you!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Could you also have a look at HADOOP-7117? ...

          Will do.

          Show
          Tsz Wo Nicholas Sze added a comment - > Could you also have a look at HADOOP-7117 ? ... Will do.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #562 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/562/)
          HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations. Contributed by Harsh J Chouraria

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #562 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/562/ ) HDFS-1596 . Replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations. Contributed by Harsh J Chouraria
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )

            People

            • Assignee:
              Harsh J
              Reporter:
              Patrick Angeles
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development