Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3443

Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.1, 3.0.0-alpha1
    • auto-failover, ha
    • None
    • Reviewed

    Description

      Start NN
      Let NN standby services be started.
      Before the editLogTailer is initialised start ZKFC and allow the activeservices start to proceed further.

      Here editLogTailer.catchupDuringFailover() will throw NPE.

      void startActiveServices() throws IOException {
      LOG.info("Starting services required for active state");
      writeLock();
      try {
      FSEditLog editLog = dir.fsImage.getEditLog();

      if (!editLog.isOpenForWrite())

      { // During startup, we're already open for write during initialization. editLog.initJournalsForWrite(); // May need to recover editLog.recoverUnclosedStreams(); LOG.info("Catching up to latest edits from old active before " + "taking over writer role in edits logs."); editLogTailer.catchupDuringFailover(); {noformat}

      2012-05-18 16:51:27,585 WARN org.apache.hadoop.ipc.Server: IPC Server Responder, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from XX.XX.XX.55:58003: output error
      2012-05-18 16:51:27,586 WARN org.apache.hadoop.ipc.Server: IPC Server handler 8 on 8020, call org.apache.hadoop.ha.HAServiceProtocol.transitionToActive from XX.XX.XX.55:58004: error: java.lang.NullPointerException
      java.lang.NullPointerException
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:602)
      at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1287)
      at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
      at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:63)
      at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1219)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:978)
      at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
      at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:3633)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
      2012-05-18 16:51:27,586 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 8020 caught an exception
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
      at org.apache.hadoop.ipc.Server.channelWrite(Server.java:2092)
      at org.apache.hadoop.ipc.Server.access$2000(Server.java:107)
      at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:930)
      at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:994)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1738)

      
      

      Attachments

        1. HDFS-3443-007.patch
          36 kB
          Vinayakumar B
        2. HDFS-3443-006.patch
          36 kB
          Vinayakumar B
        3. HDFS-3443-005.patch
          7 kB
          Vinayakumar B
        4. HDFS-3443-004.patch
          7 kB
          Vinayakumar B
        5. HDFS-3443-003.patch
          5 kB
          Vinayakumar B
        6. HDFS-3443_1.patch
          1 kB
          amith
        7. HDFS-3443_1.patch
          1 kB
          amith

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vinayakumarb Vinayakumar B Assign to me
            suja suja s
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment