[IMPALA-9154] KRPC DataStreamService threads blocked in PublishFilter

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Distributed Exec
• Labels: hang
• Target Version: Impala 3.4.0
• Epic Color: ghx-label-12
• Assignee: Fang-Yu Rao
• Reporter: Tim Armstrong
• Created: 13/Nov/19 16:36
• Updated: 20/May/20 18:36
• Resolved: 23/Jan/20 22:59

Description

I hit this on primitive_many_fragments when doing a single node perf run:

 ./bin/single_node_perf_run.py --num_impalads=1 --scale=30 --ninja --workloads=targeted-perf  --iterations=5

tan

I noticed that the query was hung and the execution threads were hung sending row batches. Then looking at the RPCz page, all of the threads were busy:

Multiple threads were stuck in UpdateFilter() - see pstack-exchange.txt. It looks like this is a deadlock bug because a KRPC thread is blocked waiting for an RPC that needs to be served by one of the limited threads from that same thread pool

Issue Links

• fixes: IMPALA-7984 Port UpdateFilter() and PublishFilter() to KRPC
• is broken by: IMPALA-7984 Port UpdateFilter() and PublishFilter() to KRPC