[IMPALA-7665] Bringing up stopped statestore causes queries to fail

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.3.0
• Component/s: Distributed Exec
• Labels: query-lifecycle statestore
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-5
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 05/Oct/18 00:11
• Updated: 03/Jun/19 16:36
• Resolved: 29/May/19 00:39

Description

I can reproduce this by running a long-running query then cycling the statestore:

tarmstrong@tarmstrong-box:~/Impala/incubator-impala$ impala-shell.sh -q "select distinct * from tpch10_parquet.lineitem"
Starting Impala Shell without Kerberos authentication
Connected to localhost:21000
Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build c486fb9ea4330e1008fa9b7ceaa60492e43ee120)
Query: select distinct * from tpch10_parquet.lineitem
Query submitted at: 2018-10-04 17:06:48 (Coordinator: http://tarmstrong-box:25000)

If I kill the statestore, the query runs fine, but if I start up the statestore again, it fails.

# In one terminal, start up the statestore
$ /home/tarmstrong/Impala/incubator-impala/be/build/latest/statestore/statestored -log_filename=statestored -log_dir=/home/tarmstrong/Impala/incubator-impala/logs/cluster -v=1 -logbufsecs=5 -max_log_files=10
# The running query then fails
WARNINGS: Failed due to unreachable impalad(s): tarmstrong-box:22001, tarmstrong-box:22002

Note that I've seen different subsets impalads reported as failed, e.g. "Failed due to unreachable impalad(s): tarmstrong-box:22001"

Issue Links

• is duplicated by: IMPALA-2626 In-flight queries fail when statestore comes back online.
• relates to: IMPALA-4137 Rolling restart of Impala