[HDDS-5033] SCM may not be able to know full port list of Datanode after Datanode is started.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: SCM HA
• Labels: pull-request-available
• Assignee: Glen Geng
• Reporter: Glen Geng
• Created: 26/Mar/21 05:26
• Updated: 01/Apr/21 09:47
• Resolved: 01/Apr/21 09:47

Description

Please check attachment.

After restart DN, the SCM may not know the full ports of that DN.

This issue can not be solved without restart SCM. The consequence is that Datanode can not participate any pipeline, and there will be continually NPE in DN.

2021-03-25 15:04:16,322 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
java.lang.NullPointerException
        at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
        at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
        at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
        at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
        at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
        at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
        at java.lang.Thread.run(Thread.java:748)
2021-03-25 15:04:16,323 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
java.lang.NullPointerException
        at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
        at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
        at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
        at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
        at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
        at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
        at java.lang.Thread.run(Thread.java:748)

 

After restart SCM, this issue gone.

The issue should be: SCMNodeManager just record the DatanodeDetails once during register.

But for DN, it won’t record the admin, server, client port into DatanodeDetails until its ratis server is up.

Thus there is contention here: if the register request is reported before ratis server is up, SCM won’t know full port list of that DN.

 

 UPDATE

public void start(String clusterId) throws IOException {
  if (!isStarted.compareAndSet(false, true)) {
    LOG.info("Ignore. OzoneContainer already started.");
    return;
  }
  LOG.info("Attempting to start container services.");
  startContainerScrub();

  replicationServer.start();
  datanodeDetails.setPort(Name.REPLICATION, replicationServer.getPort());

  writeChannel.start();
  readChannel.start();
  hddsDispatcher.init();
  hddsDispatcher.setClusterId(clusterId);
  blockDeletingService.start();
}

We are doing SCM HA test, which means the start will called multi times, and only the first SCM connection will succeed in the CAS. The second SCM connection will won't wait for writeChannel.start(); thus get a partial port list.

 

UPDATE again

It is the contention of connect to multi SCMs at DN side. We also needs add lock to DatanodeDetails.

Issue Links

• links to: GitHub Pull Request #2090