Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-208

name node should warn if only one dir is listed in dfs.name.dir

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The name node should warn that corruption may occur if only one directory is listed in the dfs.name.dir setting.

      1. HDFS-208.patch
        1 kB
        Uma Maheswara Rao G
      2. hdfs-208.patch
        1 kB
        Eli Collins

        Issue Links

          Activity

          Allen Wittenauer created issue -
          Allen Wittenauer made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Minor [ 4 ]
          Allen Wittenauer made changes -
          Link This issue is related to HADOOP-3877 [ HADOOP-3877 ]
          Hide
          Allen Wittenauer added a comment -

          The idea here is make it obvious that the admin of a production system is doing something very, very dangerous.

          Show
          Allen Wittenauer added a comment - The idea here is make it obvious that the admin of a production system is doing something very, very dangerous.
          Allen Wittenauer made changes -
          Link This issue is related to HADOOP-4080 [ HADOOP-4080 ]
          Hide
          Devaraj Das added a comment -

          Please mark it as a blocker for 0.19 if required

          Show
          Devaraj Das added a comment - Please mark it as a blocker for 0.19 if required
          Devaraj Das made changes -
          Fix Version/s 0.19.0 [ 12313211 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-4081 HDFS-208
          Affects Version/s 0.18.0 [ 12312972 ]
          Component/s dfs [ 12310710 ]
          Eli Collins made changes -
          Labels newbie
          Fix Version/s 0.23.0 [ 12315571 ]
          Component/s name-node [ 12312926 ]
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.0 [ 12315571 ]
          Allen Wittenauer made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Eli Collins added a comment -

          Re-opening, seems like an easy, valuable improvement.

          Show
          Eli Collins added a comment - Re-opening, seems like an easy, valuable improvement.
          Eli Collins made changes -
          Resolution Won't Fix [ 2 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Uma Maheswara Rao G made changes -
          Assignee Uma Maheswara Rao G [ umamaheswararao ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-208.patch [ 12502638 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Provided the simple patch.
          Verified the patch manually. Below are the warn logs.

          2011-11-06 09:38:31,611 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(314)) - Only one directory configured for dfs.namenode.name.dir, chances for namespace corruption
          2011-11-06 09:38:31,612 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(319)) - Only one directory configured for dfs.namenode.edits.dir, chances for edits corruption

          Not provided the tests because these are just log additions. I feel, not required to start NN for asserting logs.

          Show
          Uma Maheswara Rao G added a comment - Provided the simple patch. Verified the patch manually. Below are the warn logs. 2011-11-06 09:38:31,611 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(314)) - Only one directory configured for dfs.namenode.name.dir, chances for namespace corruption 2011-11-06 09:38:31,612 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(319)) - Only one directory configured for dfs.namenode.edits.dir, chances for edits corruption Not provided the tests because these are just log additions. I feel, not required to start NN for asserting logs.
          Uma Maheswara Rao G made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502638/HDFS-208.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502638/HDFS-208.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1538//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Test failures are unrelated to this patch!

          Show
          Uma Maheswara Rao G added a comment - Test failures are unrelated to this patch!
          Hide
          Eli Collins added a comment -

          +1 looks good

          Show
          Eli Collins added a comment - +1 looks good
          Eli Collins made changes -
          Issue Type New Feature [ 2 ] Improvement [ 4 ]
          Fix Version/s 0.23.1 [ 12318885 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Hide
          Eli Collins added a comment -

          Minor update that's a little more explicit (warns of data loss vs nn corruption).

          Show
          Eli Collins added a comment - Minor update that's a little more explicit (warns of data loss vs nn corruption).
          Eli Collins made changes -
          Attachment hdfs-208.patch [ 12504409 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1363/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1363/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1289 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1289/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1289 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1289/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12504409/hdfs-208.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1586//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504409/hdfs-208.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1586//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          I committed this. Thanks Uma!

          Show
          Eli Collins added a comment - I committed this. Thanks Uma!
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.1 [ 12318885 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1315/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1315/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/)
          HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/ ) HDFS-208 . name node should warn if only one dir is listed in dfs.name.dir. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204119 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #666 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/666/)
          HDFS-208. Merge r1204119 from trunk to 0.23 (Revision 1298231)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298231
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #666 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/666/ ) HDFS-208 . Merge r1204119 from trunk to 0.23 (Revision 1298231) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298231 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Suresh Srinivas made changes -
          Fix Version/s 0.23.3 [ 12320052 ]
          Target Version/s 0.23.3 [ 12320052 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1926/)
          Moving HDFS-3036, HDFS-3014, HDFS-3003, HDFS-2878, HDFS-208, HDFS-2764, HDFS-2410, HDFS-2285, HDFS-2507 to 0.23.3 section (Revision 1298274)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1926/ ) Moving HDFS-3036 , HDFS-3014 , HDFS-3003 , HDFS-2878 , HDFS-208 , HDFS-2764 , HDFS-2410 , HDFS-2285 , HDFS-2507 to 0.23.3 section (Revision 1298274) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1860/)
          Moving HDFS-3036, HDFS-3014, HDFS-3003, HDFS-2878, HDFS-208, HDFS-2764, HDFS-2410, HDFS-2285, HDFS-2507 to 0.23.3 section (Revision 1298274)

          Result = ABORTED
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1860/ ) Moving HDFS-3036 , HDFS-3014 , HDFS-3003 , HDFS-2878 , HDFS-208 , HDFS-2764 , HDFS-2410 , HDFS-2285 , HDFS-2507 to 0.23.3 section (Revision 1298274) Result = ABORTED suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298274 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Arun C Murthy made changes -
          Fix Version/s 2.0.0 [ 12320353 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.3 [ 12320052 ]
          Allen Wittenauer made changes -
          Fix Version/s 0.23.3 [ 12320052 ]
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          1153d 3h 13m 1 Allen Wittenauer 02/Nov/11 17:56
          Resolved Resolved Reopened Reopened
          45m 11s 1 Eli Collins 02/Nov/11 18:41
          Reopened Reopened Patch Available Patch Available
          3d 9h 34m 1 Uma Maheswara Rao G 06/Nov/11 04:16
          Patch Available Patch Available Resolved Resolved
          13d 23h 34m 1 Eli Collins 20/Nov/11 03:50

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development