[NIFI-5361] Upon restart of NiFi, some processors show Active Threads but never actually run

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.8.0, 1.7.1
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 02/Jul/18 13:02
• Updated: 12/Jul/18 01:31
• Resolved: 02/Jul/18 20:37

Description

Upon restart, if there are a large number of processors that take a long time to start up (i.e., their @OnScheduled methods take a long time to complete), then other processors may also timeout while waiting for their @OnScheduled method to complete. It looks like this is due to the fact that when we schedule many processors we determine the 'timeout timestamp' before submitting the background task to call @OnScheduled. As a result, if several processors don't finish or take a long time to finish, then other processors may just sit, waiting for their chance to have @OnScheduled called and eventually timeout before they even have a chance to invoke the method.

When this happens, the result is that the Runnable to call @OnScheduled never is called before the corresponding Future is canceled. So the Processor is never scheduled. However, its Active Thread count has already been incremented. So it appears as if the Processor has an Active Thread but it will never make progress.

Issue Links

• links to: GitHub Pull Request #2831