Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-208

name node should warn if only one dir is listed in dfs.name.dir

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The name node should warn that corruption may occur if only one directory is listed in the dfs.name.dir setting.

      1. HDFS-208.patch
        1 kB
        Uma Maheswara Rao G
      2. hdfs-208.patch
        1 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1860/)
          Moving HDFS-3036, HDFS-3014, HDFS-3003, HDFS-2878, HDFS-208, HDFS-2764, HDFS-2410, HDFS-2285, HDFS-2507 to 0.23.3 section (Revision 1298274)

          Result = ABORTED
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1860/ ) Moving HDFS-3036 , HDFS-3014 , HDFS-3003 , HDFS-2878 , HDFS-208 , HDFS-2764 , HDFS-2410 , HDFS-2285 , HDFS-2507 to 0.23.3 section (Revision 1298274) Result = ABORTED suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1926/)
          Moving HDFS-3036, HDFS-3014, HDFS-3003, HDFS-2878, HDFS-208, HDFS-2764, HDFS-2410, HDFS-2285, HDFS-2507 to 0.23.3 section (Revision 1298274)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1926/ ) Moving HDFS-3036 , HDFS-3014 , HDFS-3003 , HDFS-2878 , HDFS-208 , HDFS-2764 , HDFS-2410 , HDFS-2285 , HDFS-2507 to 0.23.3 section (Revision 1298274) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #666 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/666/)
          HDFS-208. Merge r1204119 from trunk to 0.23 (Revision 1298231)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298231
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #666 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/666/ ) HDFS-208 . Merge r1204119 from trunk to 0.23 (Revision 1298231) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298231 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1315/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1315/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Eli Collins added a comment -

          I committed this. Thanks Uma!

          Show
          Eli Collins added a comment - I committed this. Thanks Uma!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12504409/hdfs-208.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1586//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504409/hdfs-208.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1586//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1289 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1289/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1289 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1289/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1363/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1363/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Eli Collins added a comment -

          Minor update that's a little more explicit (warns of data loss vs nn corruption).

          Show
          Eli Collins added a comment - Minor update that's a little more explicit (warns of data loss vs nn corruption).
          Hide
          Eli Collins added a comment -

          +1 looks good

          Show
          Eli Collins added a comment - +1 looks good
          Hide
          Uma Maheswara Rao G added a comment -

          Test failures are unrelated to this patch!

          Show
          Uma Maheswara Rao G added a comment - Test failures are unrelated to this patch!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502638/HDFS-208.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502638/HDFS-208.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Provided the simple patch.
          Verified the patch manually. Below are the warn logs.

          2011-11-06 09:38:31,611 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(314)) - Only one directory configured for dfs.namenode.name.dir, chances for namespace corruption
          2011-11-06 09:38:31,612 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(319)) - Only one directory configured for dfs.namenode.edits.dir, chances for edits corruption

          Not provided the tests because these are just log additions. I feel, not required to start NN for asserting logs.

          Show
          Uma Maheswara Rao G added a comment - Provided the simple patch. Verified the patch manually. Below are the warn logs. 2011-11-06 09:38:31,611 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(314)) - Only one directory configured for dfs.namenode.name.dir, chances for namespace corruption 2011-11-06 09:38:31,612 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(319)) - Only one directory configured for dfs.namenode.edits.dir, chances for edits corruption Not provided the tests because these are just log additions. I feel, not required to start NN for asserting logs.
          Hide
          Eli Collins added a comment -

          Re-opening, seems like an easy, valuable improvement.

          Show
          Eli Collins added a comment - Re-opening, seems like an easy, valuable improvement.
          Hide
          Devaraj Das added a comment -

          Please mark it as a blocker for 0.19 if required

          Show
          Devaraj Das added a comment - Please mark it as a blocker for 0.19 if required
          Hide
          Allen Wittenauer added a comment -

          The idea here is make it obvious that the admin of a production system is doing something very, very dangerous.

          Show
          Allen Wittenauer added a comment - The idea here is make it obvious that the admin of a production system is doing something very, very dangerous.

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development