[HDFS-14075] Terminate the namenode when failed to start log segment

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 14/Nov/18 06:19
• Updated: 02/Dec/18 17:35
• Resolved: 02/Dec/18 17:20

Description

2018-11-10 18:59:38,427 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Exception while edit logging: null
java.lang.NullPointerException
 at org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
 at java.lang.Thread.run(Thread.java:745)
2018-11-10 18:59:38,532 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: Exception while edit logging: null
2018-11-10 18:59:38,552 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

Before NPE Received the following Exception

INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 65110, call Call#23241 Retry#0 org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog from XXXXXXXX
java.io.IOException: Unable to start log segment 7964819: too few journals successfully started.
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:1385)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegmentAndWriteHeaderTxn(FSEditLog.java:1395)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1319)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1352)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4669)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1293)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:146)
	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12974)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684)
Caused by: java.io.IOException: starting log segment 7964819 failed for too many journals
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:412)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.startLogSegment(JournalSet.java:207)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:1383)
	... 15 more