[HDDS-4827] Datanode unable to prepare itself for finalize.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: Ozone Datanode
• Assignee: Prashant Pogde
• Reporter: Aravindan Vijayan
• Created: 16/Feb/21 14:30
• Updated: 14/Jul/21 21:12
• Resolved: 14/Jul/21 21:12

Description

2021-02-15 18:58:03,058 [Command processor thread] ERROR commandhandler.ClosePipelineCommandHandler: Can't close pipeline PipelineID=a08eb315-2da7-496f-a9af-0cafeb68f1f3
java.io.IOException: c49ad561-dbbb-46bb-b946-74de838f87cb: Group group-0CAFEB68F1F3 not found.
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:819)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ClosePipelineCommandHandler.handle(ClosePipelineCommandHandler.java:74)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:531)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.ratis.protocol.exceptions.GroupMismatchException: c49ad561-dbbb-46bb-b946-74de838f87cb: Group group-0CAFEB68F1F3 not found.
	at org.apache.ratis.server.impl.RaftServerProxy.groupRemoveAsync(RaftServerProxy.java:504)
	at org.apache.ratis.server.impl.RaftServerProxy.groupManagementAsync(RaftServerProxy.java:460)
	at org.apache.ratis.server.impl.RaftServerProxy.groupManagement(RaftServerProxy.java:440)
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:817)
	... 4 more
2021-02-15 18:58:33,026 [Command processor thread] INFO commandhandler.FinalizeNewLayoutVersionCommandHandler: Processing FinalizeNewLayoutVersionCommandHandler command.
2021-02-15 18:58:33,026 [Command processor thread] INFO commandhandler.FinalizeNewLayoutVersionCommandHandler: Finalize Upgrade called!
2021-02-15 18:58:33,026 [Command processor thread] WARN server.RaftServer: c49ad561-dbbb-46bb-b946-74de838f87cb: does not contain group: group-4D93A117906E
2021-02-15 18:58:33,026 [Command processor thread] ERROR commandhandler.ClosePipelineCommandHandler: Can't close pipeline PipelineID=0d64fe5b-b383-483c-bded-4d93a117906e
java.io.IOException: c49ad561-dbbb-46bb-b946-74de838f87cb: Group group-4D93A117906E not found.
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:819)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ClosePipelineCommandHandler.handle(ClosePipelineCommandHandler.java:74)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:531)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.ratis.protocol.exceptions.GroupMismatchException: c49ad561-dbbb-46bb-b946-74de838f87cb: Group group-4D93A117906E not found.
	at org.apache.ratis.server.impl.RaftServerProxy.groupRemoveAsync(RaftServerProxy.java:504)
	at org.apache.ratis.server.impl.RaftServerProxy.groupManagementAsync(RaftServerProxy.java:460)
	at org.apache.ratis.server.impl.RaftServerProxy.groupManagement(RaftServerProxy.java:440)
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:817)
	... 4 more

On the ozone docker compose cluster, I came across this case where a datanode is unable to finalize. Datanode had open containers before finalization.