[HDDS-1376] Datanode exits while executing client command when scmId is null

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.1
• Component/s: Ozone Datanode
• Labels: MiniOzoneChaosCluster pull-request-available
• Target Version/s: 0.5.0
• Sprint: Done
• Assignee: Hanisha Koneru
• Reporter: Mukul Kumar Singh
• Created: 03/Apr/19 11:36
• Updated: 05/Aug/19 14:45
• Resolved: 16/Apr/19 20:51

Description

Ozone Datanode exits with the following error, this happens because DN hasn't received a scmID from the SCM after registration but is processing a client command.

2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker (ExitUtils.java:terminate(133)) - Terminating with exit status 1: df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
java.io.IOException: java.lang.NullPointerException: scmId cannot be null
        at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
        at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
        at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
        at org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
        at org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
        at org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: scmId cannot be null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

Issue Links

• relates to: HDDS-830 Datanode should not start XceiverServerRatis before getting version information from SCM
• links to: GitHub Pull Request #724