[IMPALA-2626] In-flight queries fail when statestore comes back online.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: Impala 2.3.0
• Component/s: Distributed Exec
• Labels: chaosmonkey statestore usability
• Target Version: Product Backlog
• Assignee: Sailesh Mukil
• Reporter: Sailesh Mukil
• Created: 03/Nov/15 23:01
• Updated: 30/Oct/18 17:38
• Resolved: 30/Oct/18 17:38

Description

During a session, if the statestore goes down, the impalads continue execution if they have enough metadata that they've already received from the statestore prior to it's failure.

The impalads can continue execution without the statestore with the stale metadata that they posses. However, when the statestore comes back online, the first membership callback it makes to the impalad hosts, erases the "known_backends" list that the impalads have stored locally.
Therefore, in-flight queries fail(sometimes without propagating the error to the shell -> IMPALA-1325).

Solution:
Do not erase the list of "known_backends" in each impalad until the statestore has a new list to provide to the impalads.

This bug was found during initial runs of ChaosMonkey on Impala.

Issue Links

• duplicates: IMPALA-7665 Bringing up stopped statestore causes queries to fail