[DRILL-2370] Missing cancellation acknowledgements leave orphaned cancelled queries around

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.9.0
• Component/s: Execution - Flow
• Assignee: Jacques Nadeau
• Reporter: Chris Westin
• Created: 03/Mar/15 23:28
• Updated: 19/Apr/15 01:05
• Resolved: 19/Apr/15 01:05

Description

When a query is cancelled (say by Foreman.cancel()), Foreman moves to the CANCELLED state, which indicates that cancellation is in progress. Cancellation requests are sent to remote drill fragments. These should be replied to via QueryManager.statusUpdate() (inherited from FragmentStatusListener), which in turn updates Foreman. However, if any of the cancellation requests are not acknowledged (due to drillbit failure, RPC timeout, etc), then the being-cancelled query will stay that way indefinitely. We need to have a way to find such queries and force them to be cleaned up. This may require substituting stub listeners for some of the objects involved in case cancellation acknowledgements still arrive even later still – these need to be safely invokable by the RPC layer even though the query they referred to is gone.

Issue Links

• depends upon: DRILL-2245 Improve Drillbit resiliency
• is depended upon by: DRILL-2634 Profile UI: Cancelled queries are not moved to Completed Queries section
• relates to: DRILL-2762 Fix situation where node failure hangs queries.