[NIFI-9835] When node offloads, can get stuck, logging errors about a negative queue size

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0, 1.16.1
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 25/Mar/22 13:25
• Updated: 04/Apr/22 18:28
• Resolved: 25/Mar/22 17:45

Description

When a node is offloaded, we can occasionally see that it does not complete the offload and logs errors about creating a negative queue size. For example:

2022-03-24 19:43:53,919 ERROR [Load-Balanced Client Thread-1[] o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes[], Swap Queue=[0, 0 Bytes[], Swap Files=[0[], Unacknowledged=[-7, -5820 Bytes[] ] to FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes[], Swap Queue=[0, 0 Bytes[], Swap Files=[0[], Unacknowledged=[-7, -5820 Bytes[] ]
java.lang.RuntimeException: Cannot create negative queue size
    at org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:1055)
    at org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:1045)
    at org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:451)
    at org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$2.onTransactionComplete(RemoteQueuePartition.java:223)
    at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:281)
    at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
    at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 

Issue Links

• links to: GitHub Pull Request #5902