Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1690

Name Node should not process format command while it is running.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1, 0.20.2, 0.21.0, 0.23.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      Currently NameNode allows format command while it running. In this case the command is executed partially (Lock file is deleted) and an exception thrown. Because of this Name Node should be formatted after restart. This sort of cases can happen accidentally. To prevent such cases Name Node should not execute the format command partially while it is running. It can stright away throw exception/log saying, Name Node is running.

      1. HDFS-1690.patch
        5 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Hide
          Uma Maheswara Rao G added a comment -

          One option is,
          if already web server started on one port , we can just ignore/throw exception for format command.Because NN will start the webserver mandatorily right.

          Show
          Uma Maheswara Rao G added a comment - One option is, if already web server started on one port , we can just ignore/throw exception for format command.Because NN will start the webserver mandatorily right.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12473985/HDFS-1690.patch
          against trunk revision 1082263.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473985/HDFS-1690.patch against trunk revision 1082263. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/271//console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          NameNode should not format partially. If the lock file is present the format command should warn and exit. I believe this is what it does. Does it not?
          The lock file is the right way to determine that the NameNode is running, not the web server.

          Show
          Konstantin Shvachko added a comment - NameNode should not format partially. If the lock file is present the format command should warn and exit. I believe this is what it does. Does it not? The lock file is the right way to determine that the NameNode is running, not the web server.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks to Konstantin for spending time and giving comments.
          I agree with you, But the problem here is if cluster shuts down abruptly then that lock file will still present in namespace directory. In this situation if user wanted to format, he can not format the NameNode at all. Thats why i have choosen this option. Please tell me your opinion.

          Show
          Uma Maheswara Rao G added a comment - Thanks to Konstantin for spending time and giving comments. I agree with you, But the problem here is if cluster shuts down abruptly then that lock file will still present in namespace directory. In this situation if user wanted to format, he can not format the NameNode at all. Thats why i have choosen this option. Please tell me your opinion.
          Hide
          Todd Lipcon added a comment -

          In that edge condition, I think it's better to make the user manually remove the lock file before proceeding. Using the web port really seems like a hack, and for rare conditions like this, it's best to make the user be very sure of what they're doing rather than allow them to do something stupid inadvertently.

          Show
          Todd Lipcon added a comment - In that edge condition, I think it's better to make the user manually remove the lock file before proceeding. Using the web port really seems like a hack, and for rare conditions like this, it's best to make the user be very sure of what they're doing rather than allow them to do something stupid inadvertently.
          Hide
          Konstantin Shvachko added a comment -

          The purpose for the lock file "in_use.lock" is to hold a lock on that file while the NameNode is running. This is in order to prevent from accidentally starting another NN in the same directory, which can mess up the fsimage and the edits.
          When the NameNode shuts down abruptly, which it always does, as there is no other way to stop the NameNode but to kill it, the lock on the lock file will be released, and the format will be able to delete the file.
          Locking may be platform dependent, but is known to work for most traditional file systems.

          Show
          Konstantin Shvachko added a comment - The purpose for the lock file "in_use.lock" is to hold a lock on that file while the NameNode is running. This is in order to prevent from accidentally starting another NN in the same directory, which can mess up the fsimage and the edits. When the NameNode shuts down abruptly, which it always does, as there is no other way to stop the NameNode but to kill it, the lock on the lock file will be released, and the format will be able to delete the file. Locking may be platform dependent, but is known to work for most traditional file systems.
          Hide
          Konstantin Shvachko added a comment -

          Also in_use.lock is deleteOnExit. It would be good if you could investigate why it is not deleted upon exiting.

          Show
          Konstantin Shvachko added a comment - Also in_use.lock is deleteOnExit. It would be good if you could investigate why it is not deleted upon exiting.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks for your comments & sorry for the late reply.

          Lock file will be deleted for normal exiting of system ( kill <pid>)
          Lock file will not be deleted only for abrupt killing ( kill -9 <pid>)

          Below is from deleteOnExit() docs

          • Deletion will be attempted only for normal termination of the
          • virtual machine, as defined by the Java Language Specification.

          Why NN format command is deleting the in_use.lock file?
          Since hdfs format command will execute in separate JVM, it will add lock file into DeleteOnExitHook.

            FileLock tryLock() throws IOException {
                File lockF = new File(root, STORAGE_FILE_LOCK);
                lockF.deleteOnExit();
                …………
           

          As we know, Once format command completed,that particular JVM will exit. On exit of that JVM, this Hook will try to delete the files which are added to that DeleteOnExitHook. In Linux, we can delete the file even if other process using that. Delete api will return true in this case.

          I did the same Test in windows, Here we can not delete that lock file when other process using it. Just delete api will return false in this case.

          My proposal would be,
          1) if in_use.lock not present then we can allow for formatting.
          2) If in_use.lock present, then we need to check for lock.
          If it is allowing to take lock then we can allow formatting.
          If it is not allowing to take lock then we should not allow for format.

          Please let me know your opinion.

          Show
          Uma Maheswara Rao G added a comment - Thanks for your comments & sorry for the late reply. Lock file will be deleted for normal exiting of system ( kill <pid>) Lock file will not be deleted only for abrupt killing ( kill -9 <pid>) Below is from deleteOnExit() docs Deletion will be attempted only for normal termination of the virtual machine, as defined by the Java Language Specification. Why NN format command is deleting the in_use.lock file? Since hdfs format command will execute in separate JVM, it will add lock file into DeleteOnExitHook. FileLock tryLock() throws IOException { File lockF = new File(root, STORAGE_FILE_LOCK); lockF.deleteOnExit(); ………… As we know, Once format command completed,that particular JVM will exit. On exit of that JVM, this Hook will try to delete the files which are added to that DeleteOnExitHook. In Linux, we can delete the file even if other process using that. Delete api will return true in this case. I did the same Test in windows, Here we can not delete that lock file when other process using it. Just delete api will return false in this case. My proposal would be, 1) if in_use.lock not present then we can allow for formatting. 2) If in_use.lock present, then we need to check for lock. If it is allowing to take lock then we can allow formatting. If it is not allowing to take lock then we should not allow for format. Please let me know your opinion.
          Hide
          Konstantin Shvachko added a comment -

          I think we can just skip checking the presence of in_use.lock during formatting and go directly to setting the lock on that file? The effect will be the same. If file does not exist, it will create and lock it.

          I think the problem with formatting is in StorageDirectory.clearDirectory(). It calls FileUtil.fullyDelete() if curDir exists, which deletes all files in the directory, but the order of deletion is not deterministic. This probably causes the partial formatting you mentioned before. If format() first deletes in_use.lock it will fail correctly if another NN is still running. If it first delete fsimage then in_use.lock, then it will also fail, but will leave the state unrecoverable.

          So I'd propose to modify StorageDirectory.clearDirectory() to first delete file STORAGE_FILE_LOCK=in_use.lock then the rest of it. I see now it's a bug.

          Show
          Konstantin Shvachko added a comment - I think we can just skip checking the presence of in_use.lock during formatting and go directly to setting the lock on that file? The effect will be the same. If file does not exist, it will create and lock it. I think the problem with formatting is in StorageDirectory.clearDirectory() . It calls FileUtil.fullyDelete() if curDir exists, which deletes all files in the directory, but the order of deletion is not deterministic. This probably causes the partial formatting you mentioned before. If format() first deletes in_use.lock it will fail correctly if another NN is still running. If it first delete fsimage then in_use.lock, then it will also fail, but will leave the state unrecoverable. So I'd propose to modify StorageDirectory.clearDirectory() to first delete file STORAGE_FILE_LOCK=in_use.lock then the rest of it. I see now it's a bug.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Konstantin,
          Thankyou for the comments

          I think we can just skip checking the presence of in_use.lock during formatting and go directly to setting the lock on that file? The effect will be the same. If file does not exist, it will create and lock it.

          No. First itself i will check the in_use.lock file presence. After that only i am going for other conditions.

          Above mentioned proposal is:
          1) if in_use.lock not present then we can allow for formatting.

          2) If in_use.lock present, then we need to check for lock. If it is allowing to take lock then we can allow formatting.If it is not allowing to take lock then we should not allow for format.

          So I'd propose to modify StorageDirectory.clearDirectory() to first delete file STORAGE_FILE_LOCK=in_use.lock then the rest of it. I see now it's a bug.

          But here the problem could be that , As i know in Linux, always i can delete the files, no matter other process is using it or not.

          One more idea i got,
          i.e, why con't we restrict besed on PID in script file itself ?.
          It is like, how hadoop start script will restrict to start the process when it is already running.

          Show
          Uma Maheswara Rao G added a comment - Hi Konstantin, Thankyou for the comments I think we can just skip checking the presence of in_use.lock during formatting and go directly to setting the lock on that file? The effect will be the same. If file does not exist, it will create and lock it. No. First itself i will check the in_use.lock file presence. After that only i am going for other conditions. Above mentioned proposal is: 1) if in_use.lock not present then we can allow for formatting. 2) If in_use.lock present, then we need to check for lock. If it is allowing to take lock then we can allow formatting.If it is not allowing to take lock then we should not allow for format. So I'd propose to modify StorageDirectory.clearDirectory() to first delete file STORAGE_FILE_LOCK=in_use.lock then the rest of it. I see now it's a bug. But here the problem could be that , As i know in Linux, always i can delete the files, no matter other process is using it or not. One more idea i got, i.e, why con't we restrict besed on PID in script file itself ?. It is like, how hadoop start script will restrict to start the process when it is already running.
          Hide
          Todd Lipcon added a comment -

          I created a new JIRA HDFS-2877 to track the lockfile deleteOnExit issue mentioned above, since it's more specific than the format() case.

          Show
          Todd Lipcon added a comment - I created a new JIRA HDFS-2877 to track the lockfile deleteOnExit issue mentioned above, since it's more specific than the format() case.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Todd. I have seen it.

          Show
          Uma Maheswara Rao G added a comment - Thanks Todd. I have seen it.

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development