[ZEPPELIN-2116] livy interpreter needs to restart after previous session timeout in secured cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.1, 0.8.0
• Component/s: livy-interpreter
• Assignee: Jeff Zhang
• Reporter: Kshitij Badani
• Created: 15/Feb/17 01:50
• Updated: 07/Mar/17 20:56
• Resolved: 20/Feb/17 18:45

Description

1) Reduce livy.session.timeout = 300000 (5 min)
2) I started a long running job
%livy
print(1)
Thread.sleep(400000)
print(2)

This scenario runs fine.
3) After a period of inactivity, the livy session times out. I can see that on RM UI - the livy session state is FINISHED
4) Now in the same notebook, if I try to do
%livy
sc.version
It gives 404 Error

org.springframework.web.client.HttpClientErrorException: 404 Not Found
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202)
at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67)
at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at org.apache.zeppelin.livy.BaseLivyInterprereter.callRestAPI(BaseLivyInterprereter.java:398)
at org.apache.zeppelin.livy.BaseLivyInterprereter.executeStatement(BaseLivyInterprereter.java:359)
at org.apache.zeppelin.livy.BaseLivyInterprereter.interpret(BaseLivyInterprereter.java:229)
at org.apache.zeppelin.livy.BaseLivyInterprereter.interpret(BaseLivyInterprereter.java:140)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:495)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

5) Now restart the livy interpreter, and if you again try to do sc.version - it works

Issue Links

• duplicates: ZEPPELIN-1546 Zeppelin Livy Interpreter 404 Exception not caught with Kerberos
• supercedes: ZEPPELIN-1516 NPE LivySparkSQLInterpreter thrown with %livy.sql interpreter function
• links to: GitHub Pull Request #1503
• links to: GitHub Pull Request #2020
• links to: GitHub Pull Request #2021