[LENS-1169] Stopping Query Service is incorrect

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6
• Assignee: Rajat Khandelwal
• Reporter: Rajat Khandelwal
• Created: 07/Jun/16 09:05
• Updated: 25/Aug/16 11:33
• Resolved: 08/Jun/16 13:35

Description

Stopping lens server basically stops all services. For query service, the current flow is this:

• Preapre stopping:
  • Interrupt All threads (query submitter, purger, status poller etc)
• Persist state
• Stop
  • join all threads ( as mentioned above)

Each of the threads is basically running in a large loop like the following:

while (!stopped && !this.isInterrupted()) {
  try {
  } catch(InterruptedException) {
    return
  }
}

Now, interrupting a thread will cause InterruptException in the thread only when the thread is waiting/sleeping.

So, the thread can exit in two ways:

• By receiving interrupt
• If an interrupt isn't received, it'll complete the current iteration loop and then exit.

So there can be a scenario like the following (I faced such a scenario while working on LENS-904):

• Stop is called from outside
• Prepare stopping. Let's say QuerySubmitter didn't receive the interrupt and will exit after completing its current iteration.
• Persist:
  • Persist part1: Persisting driver states. e.g. HiveDriver keeps a map of query handle to hive operation handle.
• QuerySubmitter submits the query to hive, changes the state of query to LAUNCHED and exits.
• Persist:
  • Persist part 2: Persisting queries. This persists the query mentioned in the above point as LAUNCHED.

Now, on start, the states will be read back, query's state will be LAUNCHED, and HiveDriver won't have the operation handle corresponding to this query. This will cause the query to fail in next status update.

Proposed Solution:

Interrupt and join the threads before persisting in the prepareStopping phase.

Issue Links

• blocks: LENS-904 Session close should not result in queued query failures
• is related to: LENS-1173 Query getting purged before restart and getting stuck in purge loop after restart