Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2914

HA: Standby should not enter safemode when resources are low

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: ha, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.

      1. HDFS-2914-trunk.patch
        4 kB
        Vinayakumar B
      2. HDFS-2914-HDFS-1623.patch
        2 kB
        Hari Mankude
      3. HDFS-2914-HDFS-1623
        2 kB
        Hari Mankude
      4. HDFS-2914-HDFS-1623
        2 kB
        Hari Mankude
      5. hdfs-2914
        2 kB
        Hari Mankude

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1104 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1104/)
        HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1104 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1104/ ) HDFS-2914 . HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1071 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1071/)
        HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1071 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1071/ ) HDFS-2914 . HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Hide
        Vinayakumar B added a comment -

        Thanks Aaron for the review and commit.

        Show
        Vinayakumar B added a comment - Thanks Aaron for the review and commit.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2356/)
        HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2356/ ) HDFS-2914 . HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2409/)
        HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2409/ ) HDFS-2914 . HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2336 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2336/)
        HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2336 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2336/ ) HDFS-2914 . HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2.
        Hide
        Aaron T. Myers added a comment -

        +1, the latest patch looks good to me. I'm going to commit this momentarily.

        Show
        Aaron T. Myers added a comment - +1, the latest patch looks good to me. I'm going to commit this momentarily.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528715/HDFS-2914-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528715/HDFS-2914-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Attaching the Latest patch for trunk(3.0)

        Show
        Vinayakumar B added a comment - Attaching the Latest patch for trunk(3.0)
        Hide
        Vinayakumar B added a comment -

        Cancelling the recent patc on 2.0.1 branch

        Show
        Vinayakumar B added a comment - Cancelling the recent patc on 2.0.1 branch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528562/HDFS-2914-2.0.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2505//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528562/HDFS-2914-2.0.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2505//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Submitting patch for 2.0.1 branch

        Show
        Vinayakumar B added a comment - Submitting patch for 2.0.1 branch
        Hide
        Vinayakumar B added a comment -

        Attaching patch for movement of NNRM from CommonService to ActiveService

        Show
        Vinayakumar B added a comment - Attaching patch for movement of NNRM from CommonService to ActiveService
        Hide
        Vinayakumar B added a comment -

        Hi all,
        According to the current patch proposed, there is no necessity of NameNodeResourceMonitor for standby NN.

        that means StandBy will never go to SafeMode because of resource unavailability. In that case, we can start this NNRM as activeService instead of commonService.

        Am I missing something..?

        Show
        Vinayakumar B added a comment - Hi all, According to the current patch proposed, there is no necessity of NameNodeResourceMonitor for standby NN. that means StandBy will never go to SafeMode because of resource unavailability. In that case, we can start this NNRM as activeService instead of commonService. Am I missing something..?
        Hide
        Aaron T. Myers added a comment -

        Converting to top-level issue with commit of HDFS-1623.

        Show
        Aaron T. Myers added a comment - Converting to top-level issue with commit of HDFS-1623 .
        Hide
        Aaron T. Myers added a comment -

        I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason.

        Hmmmmm, I don't think this should be a problem. We currently support transitioning to the active state while the NN is in safemode, so I don't see why any services would fail to start if we were to enter safemode while transitioning to the active state.

        Regardless, even if it is possible, I think you've convinced me that it's not actually necessary.

        I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode.

        OK, that seems fine. Perhaps we could also have FSNS#startActiveServices interrupt the NameNodeResourceMonitor thread? That would guarantee that a resource check would happen promptly after transitioning to active.

        Offline Todd pointed out to me that another thing we could do would be to check for having available resources in the monitorHealth RPC call, which the failover controller can call before initiating a failover, to make sure there are available resources on the NN which we want to failover to. That should probably be done in a separate JIRA, though.

        Show
        Aaron T. Myers added a comment - I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason. Hmmmmm, I don't think this should be a problem. We currently support transitioning to the active state while the NN is in safemode, so I don't see why any services would fail to start if we were to enter safemode while transitioning to the active state. Regardless, even if it is possible, I think you've convinced me that it's not actually necessary. I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode. OK, that seems fine. Perhaps we could also have FSNS#startActiveServices interrupt the NameNodeResourceMonitor thread? That would guarantee that a resource check would happen promptly after transitioning to active. Offline Todd pointed out to me that another thing we could do would be to check for having available resources in the monitorHealth RPC call, which the failover controller can call before initiating a failover, to make sure there are available resources on the NN which we want to failover to. That should probably be done in a separate JIRA, though.
        Hide
        Jitendra Nath Pandey added a comment -

        Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services.

        I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason.
        I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode.

        Show
        Jitendra Nath Pandey added a comment - Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services. I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason. I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode.
        Hide
        Aaron T. Myers added a comment -

        Like I mentioned earlier, I would like to open a seperate test jira for this issue.

        OK. Please file the JIRA.

        Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira.

        Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services.

        Show
        Aaron T. Myers added a comment - Like I mentioned earlier, I would like to open a seperate test jira for this issue. OK. Please file the JIRA. Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira. Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services.
        Hide
        Hari Mankude added a comment -

        That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior.

        Like I mentioned earlier, I would like to open a seperate test jira for this issue.

        I don't think that's true. startCommonServices(...) only gets called at NN initialization, not on each failover.

        Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira.

        Show
        Hari Mankude added a comment - That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior. Like I mentioned earlier, I would like to open a seperate test jira for this issue. I don't think that's true. startCommonServices(...) only gets called at NN initialization, not on each failover. Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira.
        Hide
        Aaron T. Myers added a comment -

        Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective?

        Obviously not a big deal, I'm just always looking for ways to speed up the tests. Feel free to punt on this.

        Show
        Aaron T. Myers added a comment - Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective? Obviously not a big deal, I'm just always looking for ways to speed up the tests. Feel free to punt on this.
        Hide
        Hari Mankude added a comment -

        I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something?

        We are talking about sleeping for 10s vs changing cluster behavior for the nnresourcetracker thread to run every 1s and changing the sleep to 2s and then changing the default of the nnresourctracker thread back to 5s so that it does not distort the remaining portions of the test. Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective?

        Show
        Hari Mankude added a comment - I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something? We are talking about sleeping for 10s vs changing cluster behavior for the nnresourcetracker thread to run every 1s and changing the sleep to 2s and then changing the default of the nnresourctracker thread back to 5s so that it does not distort the remaining portions of the test. Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective?
        Hide
        Aaron T. Myers added a comment -

        I would rather leave this as is since I could easily make the problem happen with 10s sleep.

        I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something?

        Show
        Aaron T. Myers added a comment - I would rather leave this as is since I could easily make the problem happen with 10s sleep. I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something?
        Hide
        Aaron T. Myers added a comment -

        I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira.

        That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior.

        This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s.

        I don't think that's true. startCommonServices(...) only gets called at NN initialization, not on each failover.

        Show
        Aaron T. Myers added a comment - I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira. That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior. This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s. I don't think that's true. startCommonServices(...) only gets called at NN initialization, not on each failover.
        Hide
        Hari Mankude added a comment -

        Any further requirements? If not, can this be checked in?

        Show
        Hari Mankude added a comment - Any further requirements? If not, can this be checked in?
        Hide
        Hari Mankude added a comment -

        The patch file still needs to have the ".patch" extension.

        done

        Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds.

        I would rather leave this as is since I could easily make the problem happen with 10s sleep.

        Our coding conventions require the use of curly braces ("{}") even for single-line if statements.

        done

        What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I think the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior.

        I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira.

        Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this.

        This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s.

        Show
        Hari Mankude added a comment - The patch file still needs to have the ".patch" extension. done Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds. I would rather leave this as is since I could easily make the problem happen with 10s sleep. Our coding conventions require the use of curly braces ("{}") even for single-line if statements. done What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I think the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior. I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira. Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this. This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s.
        Hide
        Aaron T. Myers added a comment -

        Hey Hari, a few more comments:

        1. The patch file still needs to have the ".patch" extension.
        2. Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds.
        3. Our coding conventions require the use of curly braces ("{}") even for single-line if statements.
        4. What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I think the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior.
        5. Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this.
        Show
        Aaron T. Myers added a comment - Hey Hari, a few more comments: The patch file still needs to have the ".patch" extension. Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds. Our coding conventions require the use of curly braces ("{}") even for single-line if statements. What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I think the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior. Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this.
        Hide
        Hari Mankude added a comment -

        New patch attached with test fixes from comments

        Show
        Hari Mankude added a comment - New patch attached with test fixes from comments
        Hide
        Aaron T. Myers added a comment -

        Hey Hari, a few comments:

        1. Is the "while (nn1.getNamesystem().isInSafeMode())" actually necessary? I would think this would be covered by the "cluster.waitActive()" call above. If not, then that's a bug in the minicluster that should be fixed.
        2. It doesn't seem like this test would necessarily reliably fail without the fix, given that the NNResourceChecker thread runs asynchronously and only every few seconds.
        3. Rather than doing "assertEquals(false, ...)" just do "assertFalse(...)".
        Show
        Aaron T. Myers added a comment - Hey Hari, a few comments: Is the "while (nn1.getNamesystem().isInSafeMode())" actually necessary? I would think this would be covered by the "cluster.waitActive()" call above. If not, then that's a bug in the minicluster that should be fixed. It doesn't seem like this test would necessarily reliably fail without the fix, given that the NNResourceChecker thread runs asynchronously and only every few seconds. Rather than doing "assertEquals(false, ...)" just do "assertFalse(...)".
        Hide
        Aaron T. Myers added a comment -

        Sure, changed name.

        Thanks.

        Test added.

        Cool. I'll take a look later, unless someone beats me to it.

        Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch.

        I challenge you to find a benchmark that produces meaningfully improved results with only running the check every 60 seconds instead of every 5.

        Show
        Aaron T. Myers added a comment - Sure, changed name. Thanks. Test added. Cool. I'll take a look later, unless someone beats me to it. Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch. I challenge you to find a benchmark that produces meaningfully improved results with only running the check every 60 seconds instead of every 5.
        Hide
        Hari Mankude added a comment -

        Uploaded patch

        @Aaron

        1. Sure, changed name.
        2. Test added.
        3. Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch.

        Show
        Hari Mankude added a comment - Uploaded patch @Aaron 1. Sure, changed name. 2. Test added. 3. Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch.
        Hide
        Aaron T. Myers added a comment -

        Hey Hari, a couple quick comments:

        1. When uploading a patch, please name the file according to the guidelines specified in http://wiki.apache.org/hadoop/HowToContribute. Specifically, this patch file should be named "HDFS-2914-HDFS-1623.patch"
        2. Please add a test for this new behavior.
        3. I don't think there's any need to change the resource check interval. Checking available disk space isn't a heavy-weight operation.
        Show
        Aaron T. Myers added a comment - Hey Hari, a couple quick comments: When uploading a patch, please name the file according to the guidelines specified in http://wiki.apache.org/hadoop/HowToContribute . Specifically, this patch file should be named " HDFS-2914 - HDFS-1623 .patch" Please add a test for this new behavior. I don't think there's any need to change the resource check interval. Checking available disk space isn't a heavy-weight operation.
        Hide
        Hari Mankude added a comment -

        Uploaded patch. In the patch, I increased the default value of RESOURCE_CHECK_INTERVAL from 5secs to 60secs. 5secs seems to be excessive for this thread to run. If there are issues, I can remove the fix or modify the fix if there is a better number.

        Show
        Hari Mankude added a comment - Uploaded patch. In the patch, I increased the default value of RESOURCE_CHECK_INTERVAL from 5secs to 60secs. 5secs seems to be excessive for this thread to run. If there are issues, I can remove the fix or modify the fix if there is a better number.
        Hide
        Aaron T. Myers added a comment -

        That sounds fine to me. Since it's not really specific to the shared edits dir per se, let's change the JIRA title to something like "HA: The standby should not enter safemode automatically when resources are low".

        Show
        Aaron T. Myers added a comment - That sounds fine to me. Since it's not really specific to the shared edits dir per se, let's change the JIRA title to something like "HA: The standby should not enter safemode automatically when resources are low".
        Hide
        Hari Mankude added a comment -

        I think Jitendra's comment makes sense. Standby will not be writing to edits directory. Making a resource check a nop should be ok for standby.

        Show
        Hari Mankude added a comment - I think Jitendra's comment makes sense. Standby will not be writing to edits directory. Making a resource check a nop should be ok for standby.
        Hide
        Jitendra Nath Pandey added a comment -

        Standby doesn't need to enter safe mode because it is not writing any transactions anyway. When it transitions to active, that's when a check for available resources to write logs should be performed.

        Show
        Jitendra Nath Pandey added a comment - Standby doesn't need to enter safe mode because it is not writing any transactions anyway. When it transitions to active, that's when a check for available resources to write logs should be performed.
        Hide
        Uma Maheswara Rao G added a comment -

        I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes."

        Yes, this sounds good. As NameNodeResourceChecker moved the system into safemode on some condition, should be its responsibility to take out of the safemode whenever system is out of that condition.

        Show
        Uma Maheswara Rao G added a comment - I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes." Yes, this sounds good. As NameNodeResourceChecker moved the system into safemode on some condition, should be its responsibility to take out of the safemode whenever system is out of that condition.
        Hide
        Aaron T. Myers added a comment -

        The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour?

        I think we probably do. If the NFS mount is flaky, we've got bigger problems than just the NN being moved into SM.

        The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts.

        I guess the admin should configure some alerts for the NN being in SM, then.

        But regardless, I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes." Such a change would not be specific to the HA branch, however, and should probably be done on trunk.

        Show
        Aaron T. Myers added a comment - The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour? I think we probably do. If the NFS mount is flaky, we've got bigger problems than just the NN being moved into SM. The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts. I guess the admin should configure some alerts for the NN being in SM, then. But regardless, I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes." Such a change would not be specific to the HA branch, however, and should probably be done on trunk.
        Hide
        Hari Mankude added a comment -

        Hi Aaron,
        The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour? The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts.

        Show
        Hari Mankude added a comment - Hi Aaron, The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour? The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts.
        Hide
        Aaron T. Myers added a comment -

        Hey Hari, per the discussion on HDFS-1594, it is by design that the NN does not automatically leave SM even after resources become available again. In order to leave SM, the admin can run `hdfs dfsadmin -safemode leave', even while the NN is in the standby state.

        Show
        Aaron T. Myers added a comment - Hey Hari, per the discussion on HDFS-1594 , it is by design that the NN does not automatically leave SM even after resources become available again. In order to leave SM, the admin can run `hdfs dfsadmin -safemode leave', even while the NN is in the standby state.
        Hide
        Hari Mankude added a comment -

        When shared edits dir is bounced, df will return space of zero. Since shared is required dir, standby nn will enter into safe mode.

        2012-02-08 01:08:19,850 WARN namenode.NameNodeResourceChecker (NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on volume 'nfs directory' is 0, which is below the configured reserved amount 104857600
        2012-02-08 01:08:19,853 WARN namenode.FSNamesystem (FSNamesystem.java:run(3095)) - NameNode low on available disk space. Entering safe mode.

        The fix could be trivial enough to exit safe mode when shared resources become available for standby NN.

        Show
        Hari Mankude added a comment - When shared edits dir is bounced, df will return space of zero. Since shared is required dir, standby nn will enter into safe mode. 2012-02-08 01:08:19,850 WARN namenode.NameNodeResourceChecker (NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on volume 'nfs directory' is 0, which is below the configured reserved amount 104857600 2012-02-08 01:08:19,853 WARN namenode.FSNamesystem (FSNamesystem.java:run(3095)) - NameNode low on available disk space. Entering safe mode. The fix could be trivial enough to exit safe mode when shared resources become available for standby NN.

          People

          • Assignee:
            Vinayakumar B
            Reporter:
            Hari Mankude
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development