[DRILL-4296] Query hangs in CANCELLATION_REQUESTED when cancelled after it starts returning results

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Component/s: Execution - Flow

Environment

commit.id=c9dbfbd
2 nodes with 32 cores and 32GB of max direct memory for drill

• Assignee: Unassigned
• Reporter: Abdel Hakim Deneche
• Reviewer: Khurram Faraaz
• Created: 21/Jan/16 18:43
• Updated: 31/Aug/17 06:58
• Resolved: 31/Aug/17 06:58

Description

After running the following queries (it's the same reproduction from DRILL-2274):

set planner.memory.max_query_memory_per_node=8589934592;
select sub1.uid from `all2274.json` sub1 inner join `all2274.json` sub2 on sub1.uid = sub2.uid order by sub1.uid;

After the query starts returning results, I cancelled the query from sqlline. This caused the query to hang in a CANCELLATION_REQUESTED state.

Looking at jstack (attached) the root fragment is blocked waiting for Ack from the client.

The the foreman node (which also runs Zookeeper) runs out of disk space once the query finishes spilling, which seems to contribute to the issue. Once I changed the spill directory to nfs I no longer so the issue.

Issue Links

• is related to: DRILL-5599 Notify StatusHandlerListener that batch sending has failed even if channel is still open