[DRILL-4676] Foreman.moveToState can block forever if called by the foreman thread while the query is still being setup

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: Execution - Flow
• Assignee: Abdel Hakim Deneche
• Reporter: Abdel Hakim Deneche
• Reviewer: Chun Chang
• Created: 15/May/16 15:03
• Updated: 03/Jun/16 18:51
• Resolved: 19/May/16 03:41

Description

When the query is being setup, foreman has a special CountDownLatch that blocks rpc threads from delivering external events, this latch is unblocked at the end of the query setup.
In some cases though, when the foreman is submitting remote fragments, a failure in RpcBus.send() causes an exception to be thrown that is reported to
Foreman.FragmentSubmitListener and blocks in the CountDownLatch. This causes the foreman thread to block forever, and can rpc threads to be blocked too.
This seems to happen more frequently at a high concurrency load, and also can prevent clients from connecting to the Drillbits.

Issue Links

• links to: GitHub Pull Request #503
• links to: GitHub Pull Request #503
• links to: GitHub Pull Request #503
• links to: GitHub Pull Request #503
• links to: GitHub Pull Request #503