[BOOKKEEPER-219] Active Namenode shutdown after throwing 'java.io.IOException: BookKeeper error during close'.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 4.0.0
• Component/s: bookkeeper-client

Environment

10.18.40.45: Active namenode
10.18.40.91: Backup namenode
10.18.52.157 : one zookeeper, three bookkeeper

• Assignee: Unassigned
• Reporter: Surendra Singh Lilhore
• Created: 18/Apr/12 06:42
• Updated: 08/May/12 13:30
• Resolved: 19/Apr/12 03:42

Description

scenario:

1. I am started Active namenode and backup namenode.
2. write some file.

Result:

ANN throwing following Error

 
2012-04-18 10:46:04,001 WARN org.apache.bookkeeper.client.LedgerHandle: Conditional write failed: BADVERSION
2012-04-18 10:46:04,001 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: finalize log segment 1, 8 failed for required journal (JournalAndStream(mgr=org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager@1ec3362f, stream=org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream@221a5770))
java.io.IOException: BookKeeper error during close
	at org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream.close(BookKeeperEditLogOutputStream.java:90)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:79)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:180)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:322)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:176)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:925)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:855)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:971)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4092)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:714)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:113)
	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8068)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:417)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:891)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1661)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1657)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1205)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1655)
Caused by: org.apache.bookkeeper.client.BKException$ZKException
	at org.apache.bookkeeper.client.BKException.create(BKException.java:64)
	at org.apache.bookkeeper.client.LedgerHandle.close(LedgerHandle.java:216)
	at org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream.close(BookKeeperEditLogOutputStream.java:86)
	... 19 more

BNN throwing following Error

 
2012-04-18 10:39:09,421 ERROR org.apache.bookkeeper.client.PendingReadOp: Error: No such entry while reading entry: 1 ledgerId: 5 from bookie: /10.18.52.157:3183
2012-04-18 10:39:09,423 ERROR org.apache.bookkeeper.client.PendingReadOp: Error: No such entry while reading entry: 1 ledgerId: 5 from bookie: /10.18.52.157:3181
2012-04-18 10:39:09,457 WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edits tailer failed to find any streams. Will try again later.
java.io.IOException: No ledger for fromTxnId -12344 found.
	at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.getInputStream(BookKeeperJournalManager.java:329)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:246)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1100)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:206)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$700(EditLogTailer.java:59)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:318)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:276)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:293)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:504)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:289)

Issue Links

• duplicates: HDFS-3058 HA: Bring BookKeeperJournalManager up to date with HA changes