[NIFI-8667] Deadlock when enabling Controller Service and all Referencing Components

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.14.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 07/Jun/21 21:57
• Updated: 08/Jun/21 16:00
• Resolved: 08/Jun/21 16:00

Description

I have a flow configured with an AvroSchemaRegistry. I have a JSON Reader and a JSON Writer both referencing it. I then have a few processors referencing both the reader and the writer.
When I click Enable on the AvroSchemaRegistry and choose to enable all referencing components, I encountered a deadlock. The threads involved were:

"Timer-Driven Process Thread-9" Id=80 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@32a7e726 ** DEADLOCKED THREAD **
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.apache.nifi.controller.service.ServiceStateTransition.getState(ServiceStateTransition.java:125)
at org.apache.nifi.controller.service.StandardControllerServiceNode.getState(StandardControllerServiceNode.java:363)
at org.apache.nifi.controller.AbstractComponentNode.validateReferencedControllerServices(AbstractComponentNode.java:719)
at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:611)
at org.apache.nifi.controller.StandardProcessorNode.computeValidationErrors(StandardProcessorNode.java:1054)
at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:574)
at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:588)
at org.apache.nifi.controller.service.ServiceStateTransition.enable(ServiceStateTransition.java:83)
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:460)

• waiting on java.util.concurrent.atomic.AtomicBoolean@126cb602
  at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  Number of Locked Synchronizers: 2
• java.util.concurrent.ThreadPoolExecutor$Worker@6492d91f
• java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@b052adc

"Timer-Driven Process Thread-8" Id=78 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@b052adc ** DEADLOCKED THREAD **
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.apache.nifi.controller.service.ServiceStateTransition.getState(ServiceStateTransition.java:125)
at org.apache.nifi.controller.service.StandardControllerServiceNode.getState(StandardControllerServiceNode.java:363)
at org.apache.nifi.controller.AbstractComponentNode.validateReferencedControllerServices(AbstractComponentNode.java:719)
at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:611)
at org.apache.nifi.controller.StandardProcessorNode.computeValidationErrors(StandardProcessorNode.java:1054)
at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:574)
at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:588)
at org.apache.nifi.controller.service.ServiceStateTransition.enable(ServiceStateTransition.java:83)
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:460)

• waiting on java.util.concurrent.atomic.AtomicBoolean@2cdc4681
  at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  Number of Locked Synchronizers: 2
• java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@32a7e726
• java.util.concurrent.ThreadPoolExecutor$Worker@7814cc76

Issue Links

• links to: GitHub Pull Request #5134