[IMPALA-56] Async query execution hangs when WAIT debug action set in CLOSE or PREPARE of plan node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 0.5
• Fix Version/s: Impala 1.0
• Assignee: Alan Choi
• Reporter: Lenni Kuff
• Created: 08/Feb/13 18:01
• Updated: 18/Apr/13 00:06
• Resolved: 18/Apr/13 00:06

Description

Async queries will hang when a 'WAIT' debug action set in 'PREPARE', 'CLOSE' phase of a plan node.

For example, this will cause the query to wait in the CLOSE phase of node id 7 (a SORT_NODE in this case, but it doesn't seem to matter).

-- Debug Action
set debug_action=7:CLOSE:WAIT;

-- Query
select a.int_col, count(b.int_col) int_sum from hbasealltypesagg a
join
  (select * from alltypes_seq_def
   where year=2009 and month=1 order by int_col limit 2500
   union all
   select * from alltypes_seq_def
   where year=2009 and month=2 limit 3000) b
on (a.int_col = b.int_col)
group by a.int_col
order by int_sum
limit 200

The query is execution is supposed to be async over beeswax. Instead of returning "immediately", as expected, the query hangs.

# Create the impala service client
transport.open()
protocol = TBinaryProtocol.TBinaryProtocol(transport)
imp_service = ImpalaService.Client(protocol)

# Issue the async query
query = BeeswaxService.Query()
query.query = query_string
query.configuration = self.__options_to_string_list() # contains the debug_action
imp_service.query(query,)

Stack trace from attaching to impalad:

#0  0x00007fb04c7d6d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fb04c9f5fdb in boost::thread::join() () from /usr/lib/libboost_thread.so.1.46.1
#2  0x0000000000be62ab in boost::thread_group::join_all (this=0x7fb0357ad760)
    at /usr/include/boost/thread/detail/thread_group.hpp:74
#3  0x0000000000be5ab3 in impala::ParallelExecutor::Exec(boost::function<impala::Status (void*)>, void**, int)
    (function=..., args=0x4c31e58, num_args=1)
    at /home/lskuff/dev/Impala/be/src/runtime/parallel-executor.cc:32
#4  0x0000000000b94bf8 in impala::Coordinator::Exec (this=0x56cc000, query_id=..., request=0x3dd8d50, 
    query_options=...) at /home/lskuff/dev/Impala/be/src/runtime/coordinator.cc:410
#5  0x0000000000a43bde in impala::ImpalaServer::QueryExecState::Exec (this=0x3dd8800, 
    exec_request=0x7fb0357ae450) at /home/lskuff/dev/Impala/be/src/service/impala-server.cc:154
#6  0x0000000000a4f2dd in impala::ImpalaServer::ExecuteInternal (this=0x37bca80, request=..., session_key=..., 
    registered_exec_state=0x7fb0357ae77f, exec_state=0x7fb0357ae840)
    at /home/lskuff/dev/Impala/be/src/service/impala-server.cc:938
#7  0x0000000000a4ee78 in impala::ImpalaServer::Execute (this=0x37bca80, request=..., session_key=..., 
    exec_state=0x7fb0357ae840) at /home/lskuff/dev/Impala/be/src/service/impala-server.cc:894
#8  0x0000000000ac1392 in impala::ImpalaServer::query (this=0x37bca80, query_handle=..., query=...)
    at /home/lskuff/dev/Impala/be/src/service/impala-beeswax-server.cc:148
#9  0x0000000000c721f0 in beeswax::BeeswaxServiceProcessor::process_query (this=0x28337c0, seqid=0, 
    iprot=0x5430f80, oprot=0x5430f40, callContext=0x28062f0)




#1  0x00007fb04ab4d6dc in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x0000000000cf8125 in impala::ExecNode::ExecDebugAction (this=0x5d36000, 
    phase=impala::TExecNodePhase::CLOSE) at /home/lskuff/dev/Impala/be/src/exec/exec-node.cc:364
#3  0x0000000000cf60e7 in impala::ExecNode::Close (this=0x5d36000, state=0x56c3340)
    at /home/lskuff/dev/Impala/be/src/exec/exec-node.cc:89
#4  0x0000000000d60314 in impala::TopNNode::Close (this=0x5d36000, state=0x56c3340)
    at /home/lskuff/dev/Impala/be/src/exec/topn-node.cc:144