Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4238

[HA] Standby namenode should not do purging of shared storage edits.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.2-alpha
    • Fix Version/s: 3.0.0, 2.0.3-alpha
    • Component/s: ha
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This happened in our cluster,

      >> Standby NN was keep doing checkpoint every one hour and uploading to Active NN was continuously failing due to some kerberos issue and nobody noticed this, since Active was servicing properly.

      >> Active NN was up for long time with fsimage having very least transaction.

      >> Standby NN has saved the checkpoint in its name dir and purged the txns > 1000000 from shared storage ( includes edits which are not present in Active NN's fsimage)

      >> After some time Active NN is restarted and StandBy NN switched to Active.

      Now current Standby not able to load any edits from shared storage, as expected edits are not present in shared storage. Its keep running idle.

      So editLog.purgeLogsOlderThan(purgeLogsFrom); always should be called from Active NameNode.

      1. hdfs-4238.txt
        6 kB
        Todd Lipcon
      2. hdfs-4238.txt
        7 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        Hi Vinay. I agree it's a bug if the SBN is purging edits from the shared storage. However, I don't follow the scenario you mentioned:

        >> After some time Active NN is restarted and StandBy NN switched to Active.

        Now current Standby not able to load any edits from shared storage, as expected edits are not present in shared storage. Its keep running idle.

        By "current Standby", do you mean the one that was Active prior to the restart?

        Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize?

        Show
        Todd Lipcon added a comment - Hi Vinay. I agree it's a bug if the SBN is purging edits from the shared storage. However, I don't follow the scenario you mentioned: >> After some time Active NN is restarted and StandBy NN switched to Active. Now current Standby not able to load any edits from shared storage, as expected edits are not present in shared storage. Its keep running idle. By "current Standby", do you mean the one that was Active prior to the restart? Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize?
        Hide
        liaowenrui added a comment -

        By "current Standby", do you mean the one that was Active prior to the restart?

        lwr:yes

        Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize?

        lwr:this is only manual model.we hope namenode auto restoration itself

        Show
        liaowenrui added a comment - By "current Standby", do you mean the one that was Active prior to the restart? lwr:yes Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize? lwr:this is only manual model.we hope namenode auto restoration itself
        Hide
        Vinayakumar B added a comment -

        Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize

        Yes, I agree there are no permanent dataloss.

        Show
        Vinayakumar B added a comment - Just to clarify, there was no permanent data loss here - just that you had to manually copy one of the checkpoints from the new Active over to the new Standby, and restart the new Standby before you could resynchronize Yes, I agree there are no permanent dataloss.
        Hide
        Aaron T. Myers added a comment -

        Please correct me if I misunderstand this issue, but it seems the title of this JIRA isn't accurate. If I understand the scenario described, the standby NN never purged any files from the shared storage - only the active did that. The trouble is that this resulted in the standby not having sufficient transactions in the shared edits dir to be able to become active, since its fsimage was so out of date.

        If my understanding is correct, let's please change the title.

        Show
        Aaron T. Myers added a comment - Please correct me if I misunderstand this issue, but it seems the title of this JIRA isn't accurate. If I understand the scenario described, the standby NN never purged any files from the shared storage - only the active did that. The trouble is that this resulted in the standby not having sufficient transactions in the shared edits dir to be able to become active, since its fsimage was so out of date. If my understanding is correct, let's please change the title.
        Hide
        liaowenrui added a comment -

        the standby NN never purged any files from the shared storage - only the active did that

        lwr:this isn't accurate.standby nn purged editlog from the share storage in StandbyCheckpointer. i will changed it to only the active did that for this issue

        Show
        liaowenrui added a comment - the standby NN never purged any files from the shared storage - only the active did that lwr:this isn't accurate.standby nn purged editlog from the share storage in StandbyCheckpointer. i will changed it to only the active did that for this issue
        Hide
        Aaron T. Myers added a comment -

        lwr:this isn't accurate.standby nn purged editlog from the share storage in StandbyCheckpointer. i will changed it to only the active did that for this issue

        I'm sorry, but I still don't understand. Are you saying that the Standby NN actually performed the deletions of files from the shared edits dir? Or that it triggered the removals at checkpoint time? Can you perhaps point to the line(s) of code you're referring to in the StandbyCheckpointer?

        Show
        Aaron T. Myers added a comment - lwr:this isn't accurate.standby nn purged editlog from the share storage in StandbyCheckpointer. i will changed it to only the active did that for this issue I'm sorry, but I still don't understand. Are you saying that the Standby NN actually performed the deletions of files from the shared edits dir? Or that it triggered the removals at checkpoint time? Can you perhaps point to the line(s) of code you're referring to in the StandbyCheckpointer?
        Hide
        Vinayakumar B added a comment -

        Hi Aaron,
        Yes, its the standby who purged the edits from shared storage. I will describe it as below.

        1. NN1 was Active and NN2 was Standby.
        2. NN2 was doing the checkpoint every one hour. Every time saving namespace was success, but uploading to Active was failing due to some security issue. This was continued for long time. As part of saveNameSpace() in NN2, edits from shared storage purged, which are not present in NN1's fsimage. At this time Active was having old fsimage itself, but it was running.
        3. After some time, NN1 got restarted, and NN2 became Active.
        4. The current Standby NN1, is having out of date fsimage and there is a gap between fsimage and edits in shared storage. So NN1 cannot to tailing/checkpoint.

        My point is, Standby should not do any modifications to shared storage.

        Show
        Vinayakumar B added a comment - Hi Aaron, Yes, its the standby who purged the edits from shared storage. I will describe it as below. 1. NN1 was Active and NN2 was Standby. 2. NN2 was doing the checkpoint every one hour. Every time saving namespace was success, but uploading to Active was failing due to some security issue. This was continued for long time. As part of saveNameSpace() in NN2, edits from shared storage purged, which are not present in NN1's fsimage. At this time Active was having old fsimage itself, but it was running. 3. After some time, NN1 got restarted, and NN2 became Active. 4. The current Standby NN1, is having out of date fsimage and there is a gap between fsimage and edits in shared storage. So NN1 cannot to tailing/checkpoint. My point is, Standby should not do any modifications to shared storage.
        Hide
        Aaron T. Myers added a comment -

        Aha, yep, got it. Thanks a lot for the explanation. I agree with your analysis.

        Show
        Aaron T. Myers added a comment - Aha, yep, got it. Thanks a lot for the explanation. I agree with your analysis.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12555860/hdfs-4238.txt
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3594//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555860/hdfs-4238.txt against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3594//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Woops, sorry, previous patch did not apply correctly. This one should apply on trunk.

        Show
        Todd Lipcon added a comment - Woops, sorry, previous patch did not apply correctly. This one should apply on trunk.
        Hide
        Aaron T. Myers added a comment -

        The latest patch looks good to me. +1 pending Jenkins.

        Show
        Aaron T. Myers added a comment - The latest patch looks good to me. +1 pending Jenkins.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12555873/hdfs-4238.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3596//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3596//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555873/hdfs-4238.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3596//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3596//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Thanks Todd, patch looks good to me. +1.

        Show
        Vinayakumar B added a comment - Thanks Todd, patch looks good to me. +1.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk and branch. Thanks for reporting this issue, and for the reviews.

        Show
        Todd Lipcon added a comment - Committed to trunk and branch. Thanks for reporting this issue, and for the reviews.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3088/)
        HDFS-4238. Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3088/ ) HDFS-4238 . Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1246 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1246/)
        HDFS-4238. Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1246 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1246/ ) HDFS-4238 . Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1277 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1277/)
        HDFS-4238. Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1277 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1277/ ) HDFS-4238 . Standby namenode should not do purging of shared storage edits. Contributed by Todd Lipcon. (Revision 1417651) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1417651 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development