Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3612

Single namenode image directory config warning can be improved

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Currently, if you configure the NameNode to run with just one dfs.namenode.name.dir directory, it prints:

      12/07/08 20:00:22 WARN namenode.FSNamesystem: Only one dfs.namenode.name.dir directory configured , beware data loss!

      We can improve this in a few ways as it is slightly ambiguous:

      1. Fix punctuation spacing, there's always a space after a punctuation mark but never before one.
      2. Perhaps the message is better printed with a reason of why it may cause a scare of data loss. For instance, we can print "Detected a single storage directory in dfs.namenode.name.dir configuration. Beware of dataloss due to lack of redundant storage directories" or so.
      1. hdfs3612.txt
        1 kB
        Andy Isaacson
      2. hdfs3612-2.txt
        2 kB
        Andy Isaacson

        Activity

        Harsh J created issue -
        Harsh J made changes -
        Field Original Value New Value
        Labels newbie
        Hide
        Harsh J added a comment -

        I also noticed that the same warning is printed twice instead of just once:

        12/07/08 20:00:22 WARN namenode.FSNamesystem: Only one dfs.namenode.name.dir directory configured , beware data loss!
        12/07/08 20:00:22 WARN namenode.FSNamesystem: Only one dfs.namenode.edits.dir directory configured , beware data loss!
        

        This should not happen and it should be printed only once. Lets avoid redundant logs.

        Show
        Harsh J added a comment - I also noticed that the same warning is printed twice instead of just once: 12/07/08 20:00:22 WARN namenode.FSNamesystem: Only one dfs.namenode.name.dir directory configured , beware data loss! 12/07/08 20:00:22 WARN namenode.FSNamesystem: Only one dfs.namenode.edits.dir directory configured , beware data loss! This should not happen and it should be printed only once. Lets avoid redundant logs.
        Hide
        Andy Isaacson added a comment -

        Harsh - they're slightly different warnings, about name.dir and edits.dir respectively. Since the user needs to take two different actions to fix the problem, I think two warnings is reasonable.

        As an admin, I'd be annoyed to get just the name.dir error, grumble, fix it, restart, get the edits.dir error, gruble louder, fix it, restart ...

        Show
        Andy Isaacson added a comment - Harsh - they're slightly different warnings, about name.dir and edits.dir respectively. Since the user needs to take two different actions to fix the problem, I think two warnings is reasonable. As an admin, I'd be annoyed to get just the name.dir error, grumble, fix it, restart, get the edits.dir error, gruble louder, fix it, restart ...
        Hide
        Andy Isaacson added a comment -

        Improved message:

        Only one dfs.namenode.name.dir directory configured.  Beware of dataloss due to lack of redundant storage directories!
        Only one dfs.namenode.edits.dir directory configured.  Beware of dataloss due to lack of redundant storage directories!
        

        Attaching proposed patch.

        Show
        Andy Isaacson added a comment - Improved message: Only one dfs.namenode.name.dir directory configured. Beware of dataloss due to lack of redundant storage directories! Only one dfs.namenode.edits.dir directory configured. Beware of dataloss due to lack of redundant storage directories! Attaching proposed patch.
        Andy Isaacson made changes -
        Attachment hdfs3612.txt [ 12536144 ]
        Andy Isaacson made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Andy Isaacson [ adi2 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536144/hdfs3612.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        org.apache.hadoop.hdfs.server.common.TestJspHelper

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2801//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2801//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536144/hdfs3612.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.common.TestJspHelper +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2801//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2801//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Andy,

        Yeah I missed that the other said edits, my bad.

        Your fixes look good to me, can you address the couple of nits if you agree?:

        1. Double space after the full-stop in the log message can be made to a single one.
        2. When printing the config "dfs.namenode.*.dir" in each log line, perhaps lets prefix a generalized role string, for clarity? Meaning something like:

        Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!

        Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!

        Show
        Harsh J added a comment - Andy, Yeah I missed that the other said edits, my bad. Your fixes look good to me, can you address the couple of nits if you agree?: Double space after the full-stop in the log message can be made to a single one. When printing the config "dfs.namenode.*.dir" in each log line, perhaps lets prefix a generalized role string, for clarity? Meaning something like: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories! Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
        Hide
        Andy Isaacson added a comment -

        double space after period

        I always type double space after period, but I agree it's irrelevant. Fixed.

        role string

        Agreed, nice improvement.

        Attaching new patch.

        Show
        Andy Isaacson added a comment - double space after period I always type double space after period, but I agree it's irrelevant. Fixed. role string Agreed, nice improvement. Attaching new patch.
        Andy Isaacson made changes -
        Attachment hdfs3612-2.txt [ 12536314 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536314/hdfs3612-2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        org.apache.hadoop.hdfs.server.common.TestJspHelper

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2811//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2811//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536314/hdfs3612-2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.common.TestJspHelper +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2811//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2811//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Tests are failing without patch applied and the tests appear unrelated to the change.

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hdfs.server.common.TestJspHelper
        Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.737 sec <<< FAILURE!
        Running org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<< FAILURE!
        
        Results :
        
        Failed tests:   testCheckpointNode(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit
          testBackupNodeTailsEdits(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit
          testBackupNode(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit
        
        Tests in error: 
          testGetUgi(org.apache.hadoop.hdfs.server.common.TestJspHelper)
        
        Tests run: 5, Failures: 3, Errors: 1, Skipped: 0
        

        +1, committing. Thank you Andy!

        Show
        Harsh J added a comment - Tests are failing without patch applied and the tests appear unrelated to the change. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.common.TestJspHelper Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.737 sec <<< FAILURE! Running org.apache.hadoop.hdfs.server.namenode.TestBackupNode Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<< FAILURE! Results : Failed tests: testCheckpointNode(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit testBackupNodeTailsEdits(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit testBackupNode(org.apache.hadoop.hdfs.server.namenode.TestBackupNode): Test resulted in an unexpected exit Tests in error: testGetUgi(org.apache.hadoop.hdfs.server.common.TestJspHelper) Tests run: 5, Failures: 3, Errors: 1, Skipped: 0 +1, committing. Thank you Andy!
        Harsh J made changes -
        Target Version/s 2.0.1-alpha [ 12321440 ]
        Harsh J made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Hide
        Harsh J added a comment -

        Committed to branch-2 and trunk. Thank you for your contribution Andy!

        Show
        Harsh J added a comment - Committed to branch-2 and trunk. Thank you for your contribution Andy!
        Harsh J made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Target Version/s 2.0.1-alpha [ 12321440 ]
        Fix Version/s 2.0.1-alpha [ 12321440 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2527 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2527/)
        HDFS-3612. Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2527 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2527/ ) HDFS-3612 . Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2461 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2461/)
        HDFS-3612. Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2461 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2461/ ) HDFS-3612 . Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2480 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2480/)
        HDFS-3612. Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2480 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2480/ ) HDFS-3612 . Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/)
        HDFS-3612. Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059)

        Result = FAILURE
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/ ) HDFS-3612 . Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/)
        HDFS-3612. Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059)

        Result = SUCCESS
        harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/ ) HDFS-3612 . Single namenode image directory config warning can be improved. Contributed by Andy Isaacson. (harsh) (Revision 1361059) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361059 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322472 ]
        Fix Version/s 2.1.0-alpha [ 12321440 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Andy Isaacson
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development