[LENS-1286] Handle Lens server restart

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7
• Component/s: server
• Assignee: Lavkesh Lahngir
• Reporter: Lavkesh Lahngir
• Created: 24/Aug/16 07:37
• Updated: 30/Aug/17 08:00
• Resolved: 14/Sep/16 11:34

Description

Scheduled events which are generated by AlarmService for scheduling are processed by a SchedulerEventListener thread.
These steps are performed after receiving a scheduled event:
1. Get the job stored in the SchedulerDAO
2. Create a session with job owner and fill the session conf.
3. Create the instance if it is not a rerun otherwise get the instance from the store.
4. Create a Run instance.
5. Store instance (if not present) and run to the store.
6. Launch the query.
7. Update the run with the latest information.
8. If there is a failure in query launch then update the run status to be failed.
While EventListener is processing the event, it can be closed during the shutdown of scheduler service. Closing the thread can cause operations to be incomplete.
There are these stages where the thread can be interrupted:
1. Shutdown just after the alarm Event is fired and received:
In this case, we won't have any entry for the instance run in the store. This will cause a hole in the list of instances.
2. Shutdown before query launch.
In this case, at least we will have an entry in the table and status as NEW.
3. Shutdown after query launch.
In this case, the status won't be updated, and even if the query is launched we will have no information about the handle. When this happens the
scheduled status will remain NEW
Of all the scenarios above, the first one is kind of annoying because there is no way to generate the event from Alarm Service again if it is already triggered.
Second and third scenarios can be handled manually by killing the instance and re-running it.
Apart from the scheduler event listener thread, we have another thread which processes the query end event which works as follows:
1. Receive the event and check if this is a scheduled query.
2. Check the final status of the query and set it as the instance status.
If a shutdown happens after receiving the query end event, it will cause the status to remain as RUNNING.

Issue Links

• contains: LENS-1267 Instances which are in waiting state while restart are not getting resumed