Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-2823 SCM HA Support
  3. HDDS-5033

SCM may not be able to know full port list of Datanode after Datanode is started.

    XMLWordPrintableJSON

    Details

      Description

      Please check attachment.

      After restart DN, the SCM may not know the full ports of that DN.

      This issue can not be solved without restart SCM. The consequence is that Datanode can not participate any pipeline, and there will be continually NPE in DN.

      2021-03-25 15:04:16,322 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
      java.lang.NullPointerException
              at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
              at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
              at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
              at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
              at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
              at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
              at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
              at java.lang.Thread.run(Thread.java:748)
      2021-03-25 15:04:16,323 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
      java.lang.NullPointerException
              at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
              at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
              at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
              at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
              at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
              at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
              at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
              at java.lang.Thread.run(Thread.java:748)
      
      

       

      After restart SCM, this issue gone.

      The issue should be: SCMNodeManager just record the DatanodeDetails once during register.

      But for DN, it won’t record the admin, server, client port into DatanodeDetails until its ratis server is up.

      Thus there is contention here: if the register request is reported before ratis server is up, SCM won’t know full port list of that DN.

       

       UPDATE

      public void start(String clusterId) throws IOException {
        if (!isStarted.compareAndSet(false, true)) {
          LOG.info("Ignore. OzoneContainer already started.");
          return;
        }
        LOG.info("Attempting to start container services.");
        startContainerScrub();
      
        replicationServer.start();
        datanodeDetails.setPort(Name.REPLICATION, replicationServer.getPort());
      
        writeChannel.start();
        readChannel.start();
        hddsDispatcher.init();
        hddsDispatcher.setClusterId(clusterId);
        blockDeletingService.start();
      }
      

      We are doing SCM HA test, which means the start will called multi times, and only the first SCM connection will succeed in the CAS. The second SCM connection will won't wait for writeChannel.start(); thus get a partial port list.

       

      UPDATE again

      It is the contention of connect to multi SCMs at DN side. We also needs add lock to DatanodeDetails.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                glengeng Glen Geng
                Reporter:
                glengeng Glen Geng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: