[HDDS-4304] Close Container event can fail if pipeline is removed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0
• Component/s: SCM
• Labels: pull-request-available
• Target Version/s: 1.1.0
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 02/Oct/20 15:38
• Updated: 05/Oct/20 05:37
• Resolved: 05/Oct/20 05:37

Description

If you call `pipelineManager.finalizeAndDestroyPipeline()` with onTimeout=false, then the finalizePipeline call will result in a closeContainer event to be fired for every container on the pipeline. These are handled asynchronously.

However, immediately after that, the `destroyPipeline(...)` call is made. This will remove the pipeline details from the various maps / stores.

Then the closeContainer events get processed, and they attempt to remove the container from the pipeline. However as the pipeline has already been destroyed, this throws an exception and the close container events never get sent to the DNs:

2020-10-01 15:44:18,838 [EventQueue-CloseContainerForCloseContainerEventHandler] INFO container.CloseContainerEventHandler: Close container Event triggered for container : #2
2020-10-01 15:44:18,842 [EventQueue-CloseContainerForCloseContainerEventHandler] ERROR container.CloseContainerEventHandler: Failed to close the container #2.
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=59e5ae16-f1fe-45ff-9044-dd237b0e91c6 not found
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removeContainerFromPipeline(PipelineStateMap.java:372)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removeContainerFromPipeline(PipelineStateManager.java:111)
	at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removeContainerFromPipeline(SCMPipelineManager.java:413)
	at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:352)
	at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:331)
	at org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler.onMessage(CloseContainerEventHandler.java:66)
	at org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler.Onmessage(CloseContainerEventHandler.java:45)
	at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor

The simple solution is to catch the exception and ignore it.

Issue Links

• links to: GitHub Pull Request #1471