Now It works well, the paragraph bound with Spark Interpreter is running well while the Progressbar showing the percentage .
After 5 minutes later, rerun the same paragraph. This time the paragraph's status is PENDING all the time and the Progressbar is missing.
- When RemoteInterpreter expired, TimeoutLifecycleManager will call RemoteInterpreterEventServer.unRegisterInterpreterProcess which only removes the RemoteInterpreterGroup without close it.
- When the paragraph runs again, one new RemoteInterpreterGroup is instanced which asks the SchedulerFactory for one RemoteScheduler to submit the paragraph.
- SchedulerFactory always find existed RemoteScheduler, so the previous RemoteScheduler which hold the old RemoteInterpreter returned .
- The JobStatusPoller which started by the RemoteScheduler uses the old RemoteInterpreter to get status, thus an exception was thrown and it fails.
The way to fix is simple, just add the following codes to the RemoteInterpreterEventServer.unRegisterInterpreterProcess function: