[OOZIE-3646] Possible dead-lock in SignalXCommand

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.3.0
• Assignee: Junfan Zhang
• Reporter: Junfan Zhang
• Created: 13/Dec/21 08:47
• Updated: 10/Feb/22 15:06
• Resolved: 10/Feb/22 15:04

Description

The limited thread execution mechanism aims to solve the dead-lock when all active threads are executing the SignalXCommand's invokeAll method.

Dead-lock when to happen

Assuming that Oozie CallableQueue thread pool size is 120, when all threads are executing the SignalXCommand.startForkedActions method, a deadlock occurs.
Because in SignalXCommand.startForkedActions, the code of

List<Future<ActionExecutorContext>> futures = Services.get().get(CallableQueueService.class)
                    .invokeAll(tasks);

will be sync executed, however now all callableQueue threads are busy.

Solution

1. Need to limit directly invokeAll call when the num of rest threads is less than the tasks
2. To obtain correct active threads number in callableQueue, the SignalXCommand.class lock is needed.