Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14075

Terminate the namenode when failed to start log segment

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None

      Description

      2018-11-10 18:59:38,427 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Exception while edit logging: null
      java.lang.NullPointerException
       at org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
       at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
       at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
       at java.lang.Thread.run(Thread.java:745)
      2018-11-10 18:59:38,532 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: Exception while edit logging: null
      2018-11-10 18:59:38,552 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
      
      

      Before NPE Received the following Exception

      INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 65110, call Call#23241 Retry#0 org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog from XXXXXXXX
      java.io.IOException: Unable to start log segment 7964819: too few journals successfully started.
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:1385)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegmentAndWriteHeaderTxn(FSEditLog.java:1395)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1319)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1352)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4669)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1293)
      	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:146)
      	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12974)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684)
      Caused by: java.io.IOException: starting log segment 7964819 failed for too many journals
      	at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:412)
      	at org.apache.hadoop.hdfs.server.namenode.JournalSet.startLogSegment(JournalSet.java:207)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:1383)
      	... 15 more
      

        Attachments

        1. HDFS-14075-07.patch
          9 kB
          Ayush Saxena
        2. HDFS-14075-06.patch
          9 kB
          Ayush Saxena
        3. HDFS-14075-05.patch
          9 kB
          Ayush Saxena
        4. HDFS-14075-04.patch
          9 kB
          Ayush Saxena
        5. HDFS-14075-04.patch
          9 kB
          Ayush Saxena
        6. HDFS-14075-04.patch
          9 kB
          Ayush Saxena
        7. HDFS-14075-03.patch
          8 kB
          Ayush Saxena
        8. HDFS-14075-02.patch
          5 kB
          Ayush Saxena
        9. HDFS-14075-01.patch
          1 kB
          Ayush Saxena

          Activity

            People

            • Assignee:
              ayushtkn Ayush Saxena
              Reporter:
              ayushtkn Ayush Saxena
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: