Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0.2-alpha
-
Fix Version/s: 0.23.7, 2.1.0-beta
-
Component/s: namenode
-
Labels:None
-
Target Version/s:
Description
We saw the following issue in a cluster:
- The 2NN downloads an edit log segment:
2012-10-29 12:30:57,433 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /xxxxxxx/current/edits_0000000000049136809-0000000000049176162 expecting start txid #49136809
- It fails in the middle of replay due to an OOME:
2012-10-29 12:31:21,021 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation AddOp [length=0, path=/xxxxxxxx java.lang.OutOfMemoryError: Java heap space
- Future checkpoints then fail because the prior edit log replay only got halfway through the stream:
2012-10-29 12:32:21,214 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /xxxxx/current/edits_0000000000049176163-0000000000049177224 expecting start txid #49144432 2012-10-29 12:32:21,216 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint java.io.IOException: There appears to be a gap in the edit log. We expected txid 49144432, but got txid 49176163.
Issue Links
- is related to
-
HDFS-5074
Allow starting up from an fsimage checkpoint in the middle of a segment
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Aborting definitely seems like the safest thing to do-- do we know that all transactions are applied atomically (i.e. if they fail and throw an exception in the middle, is there rollback of whatever they did to the FSImage?) I'm not clear on that point.
Good point, Colin. I definitely don't think that's a safe assumption to make, so I agree we should abort if there is an exception on the application of an edit
I've seen similar issues due to a quota bug, which I have yet to reproduce. NN allows users to go a bit over quota due to this bug. On 2NN, since updateCountForINodeWithQuota() is called every time it checkpoints, any such error is corrected, but quota violations allowed by NN will still be there. (WARN is printed on 2NN) If replaying of edits on top of this namespace state meets a quota exception, it gets into the unrecoverable loop of retry.
It seems that 2NN state needs to be reset when an exception occurs during replaying edits. Then 2NN will load everything from scratch and create an accurate state without losing any edits. We will still see WARN messages revealing bugs that cause NN state to be inaccurate.
Since 2NN retries every minute, it can cause generation of a huge number of edit files. 2NN should give up after rolling and creating new edit, say, 100 times. This is related to HDFS-280, but in this case we do not download everything all over again.
The patch keeps count of merge errors. If it's > 0, reload is done. If successful, the counter is reset. If counter reaches the threshold (i.e. N consecutive checkpointing failures due to merge errors), 2NN exits in order to prevent generation of a large number of edits by rolling frequently.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571518/hdfs-4128.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 tests included appear to have a timeout.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4019//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4019//console
This message is automatically generated.
Patch looks straightforward. A few comments:
- Default of 60 retries seems a bit high, esp. for a cluster with a large image and edits. I'd think something like the standard 3 would be sufficient because if the 2NN aborts it will get the attention faster of the admin.
- I think ">= maxRetries" should be ">" because ">=" means attempts instead of retries. Ie. Currently if I specify 1 max retry, it doesn't retry at all.
- Given the former issue, there should be a test that maxRetries is actually honored.
- Spelling police told me "occured" should be "occurred"
I agree that exceptions during merge will rarely be caused by transient errors, if any. So 3 or even 2 may be sufficient. Spelling is something that constantly evolving. But I will refrain from boldly going where speller checkers have never gone before.
The revised patch includes the improved test case that checks number of retries done before exit in addition to detecting exit.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571616/hdfs-4128.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 tests included appear to have a timeout.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4023//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4023//console
This message is automatically generated.
+1 Looks good, will commit shortly.
Committed to trunk/2/23. Thanks Kihwal!
Integrated in Hadoop-trunk-Commit #3409 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3409/)
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails in the middle (kihwal via daryn) (Revision 1452384)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452384
Files :
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointConf.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointFaultInjector.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
Integrated in Hadoop-Yarn-trunk #146 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/146/)
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails in the middle (kihwal via daryn) (Revision 1452384)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452384
Files :
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointConf.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointFaultInjector.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
Integrated in Hadoop-Hdfs-0.23-Build #544 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/544/)
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails in the middle (kihwal via daryn) (Revision 1452391)
Result = FAILURE
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452391
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
Integrated in Hadoop-Hdfs-trunk #1335 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1335/)
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails in the middle (kihwal via daryn) (Revision 1452384)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452384
Files :
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointConf.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointFaultInjector.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
The issue is that it's starting replay at the beginning of the next full segment, instead of starting with the previous half-replayed segment and skipping forward to the correct txid. We could either fix this, or change the 2NN so that if edit log replay fails, it aborts itself entirely (given such errors are generally going to just happen again on the next attempt, it's probably better to fail hard so an admin notices, instead of retrying forever)