[HDDS-1757] Use ExecutorService in OzoneManagerStateMachine

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: OM HA, Ozone Manager
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Assignee: Bharat Viswanadham
• Reporter: Bharat Viswanadham
• Created: 02/Jul/19 18:41
• Updated: 04/Jun/20 20:16
• Resolved: 02/Jul/19 23:16

Description

In the current code in applyTransaction we have 

CompletableFuture<Message> future = CompletableFuture
.supplyAsync(() -> runCommand(request, trxLogIndex)); We are using ForkJoin#commonPool.

With the current approach we have 2 issues:

1. Thread exhausts when using this common pool.
2. Not a good practice of using common pool. Found some issues in our testing by using similarly in RatisPipelineUtils.
3. OM DB's across replica can be out of sync when the apply transactions are applied in out of order.

Issue Links

• links to: GitHub Pull Request #1048