[HDDS-3117] Recon throws InterruptedException while getting new snapshot from OM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 1.0.0
• Component/s: Ozone Recon
• Target Version/s: 0.5.0
• Assignee: Aravindan Vijayan
• Reporter: Vivek Ratnavel Subramanian
• Created: 02/Mar/20 21:04
• Updated: 11/Mar/20 20:18
• Resolved: 11/Mar/20 20:18

Description

Recon throws the following exception in a cluster with 21 datanodes and 1 million keys

12:46:23.482 PMINFOOzoneManagerServiceProviderImplObtaining full snapshot from Ozone Manager
12:47:08.072 PMINFOOzoneManagerServiceProviderImplGot new checkpoint from OM : /var/lib/hadoop-ozone/recon/om/data/om.snapshot.db_1583181983482
12:47:08.072 PMINFOReconOmMetadataManagerImplCleaning up old OM snapshot db at /var/lib/hadoop-ozone/recon/om/data/om.snapshot.db_1583174381836.
12:47:08.166 PMINFOReconOmMetadataManagerImplCreated OM DB handle from snapshot at /var/lib/hadoop-ozone/recon/om/data/om.snapshot.db_1583181983482.
12:47:08.276 PMINFOOzoneManagerServiceProviderImplCalling reprocess on Recon tasks.
12:47:08.276 PMERROROzoneManagerServiceProviderImplUnable to update Recon's OM DB with new snapshot 
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.apache.hadoop.ozone.recon.tasks.ReconTaskControllerImpl.reInitializeTasks(ReconTaskControllerImpl.java:175)
	at org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.syncDataFromOM(OzoneManagerServiceProviderImpl.java:375)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #648