[ZEPPELIN-1516] NPE LivySparkSQLInterpreter thrown with %livy.sql interpreter function

• Type: Bug
• Status: Reopened
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 0.6.0, 0.6.1, 0.7.0
• Component/s: livy-interpreter
• Assignee: Jeff Zhang
• Reporter: Greg Senia
• Created: 03/Oct/16 20:10
• Updated: 31/Aug/18 00:36

Description

The LivySparkSQLInterpreter class does not correctly process the userSessionMap throwing back an NPE as show below. I determined this by adding a bunch of debug statements inside of the code before making the code operate more like LivySparkRInterpreter and LivyPySparkInterpreter. This error keeps %livy.sql from functioning in any way shape or form both on 0.6.0, 0.6.1 and 0.7.0.

LivyPySparkInterpreter code snippit:
public class LivyPySparkInterpreter extends Interpreter {

Logger LOGGER = LoggerFactory.getLogger(LivyPySparkInterpreter.class);

protected Map<String, Integer> userSessionMap;
protected LivyHelper livyHelper;

public LivyPySparkInterpreter(Properties property)

{ super(property); userSessionMap = new HashMap<>(); livyHelper = new LivyHelper(property); }

LivySparkRInterpreter code snippit:
public class LivySparkRInterpreter extends Interpreter {

Logger LOGGER = LoggerFactory.getLogger(LivySparkRInterpreter.class);

protected Map<String, Integer> userSessionMap;
private LivyHelper livyHelper;

public LivySparkRInterpreter(Properties property) { super(property); userSessionMap = new HashMap<>(); livyHelper = new LivyHelper(property); }

Broken LivySparkSQLInterpreter snippit:
public class LivySparkSQLInterpreter extends Interpreter {

Logger LOGGER = LoggerFactory.getLogger(LivySparkSQLInterpreter.class);

protected Map<String, Integer> userSessionMap;
private LivyHelper livyHelper;

public LivySparkSQLInterpreter(Properties property) {
super(property);
livyHelper = new LivyHelper(property);
userSessionMap = LivySparkInterpreter.getUserSessionMap();

Working LivySparkSQLInterpreter snippit:
public class LivySparkSQLInterpreter extends Interpreter {

Logger LOGGER = LoggerFactory.getLogger(LivySparkSQLInterpreter.class);

protected Map<String, Integer> userSessionMap;
private LivyHelper livyHelper;

public LivySparkSQLInterpreter(Properties property)

{ super(property); livyHelper = new LivyHelper(property); userSessionMap = new HashMap<>(); }

ERROR [2016-09-29 23:44:30,891] (

{pool-2-thread-2}

LivySparkSQLInterpreter.java[interpret]:119) - Exception in LivySparkSQLInterpreter while interpret
java.lang.NullPointerException
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.interpret(LivySparkSQLInterpreter.java:60)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• Dependent: ZEPPELIN-1546 Zeppelin Livy Interpreter 404 Exception not caught with Kerberos
• is superceded by: ZEPPELIN-2116 livy interpreter needs to restart after previous session timeout in secured cluster
• links to: GitHub Pull Request #1503
• links to: GitHub Pull Request #1514