[DRILL-4595] FragmentExecutor.fail() should interrupt the fragment thread to avoid possible query hangs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: Future
• Assignee: Abdel Hakim Deneche
• Reporter: Abdel Hakim Deneche
• Reviewer: Khurram Faraaz
• Created: 08/Apr/16 06:19
• Updated: 14/Sep/17 22:12
• Resolved: 14/Sep/17 22:12

Description

When a fragment fails it's assumed it will be able to close itself and send it's FAILED state to the foreman which will cancel any running fragments. FragmentExecutor.cancel() will interrupt the thread making sure those fragment don't stay blocked.
However, if a fragment is already blocked when it's fail method is called the foreman may never be notified about this and the query will hang forever. One such scenario is the following:

• generally it's a CTAS running on a large cluster (lot's of writers running in parallel)
• logs show that the user channel was closed and UserServer caused the root fragment to move to a FAILED state
• jstack shows that the root fragment is blocked in it's receiver waiting for data
• jstack also shows that ALL other fragments are no longer running, and the logs show that all of them succeeded
• the foreman waits forever for the root fragment to finish

Issue Links

• is related to: DRILL-5599 Notify StatusHandlerListener that batch sending has failed even if channel is still open