[ZEPPELIN-1159] Livy interpreter gets "404 not found" error

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.6.1, 0.7.0
• Component/s: zeppelin-interpreter
• Assignee: Unassigned
• Reporter: Yesha Vora
• Created: 13/Jul/16 00:12
• Updated: 22/Jul/16 07:03
• Resolved: 19/Jul/16 12:22

Description

Livy SparkR gets "404 not found" error

Scenario:

• Start a notebook and Run below sparkR paragraph.

  %sh
  hdfs dfs -copyFromLocal examples/src/main/resources/people.json /tmp

%livy.sparkr
sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3")
sqlContext <- sparkRSQL.init(sc)
people <- read.df(sqlContext, "/tmp/people.json", "json") 
head(people)
printSchema(people)
write.df(people, path="people.parquet", source="parquet", mode="overwrite")
============================output ==================================
Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
  age    name
1  NA Michael
2  30    Andy
3  19  Justin
root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)
NULL
Attaching package: ‘SparkR’
The following objects are masked from ‘package:stats’:
    cov, filter, lag, na.omit, predict, sd, var
The following objects are masked from ‘package:base’:
    colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
    startsWith, subset, summary, table, transform

• let the livy session expire.
• run the same paragraph again.
  %livy.sparkR fails to execute with "404 Not Found"

livy interpreter log

 INFO [2016-07-12 22:59:35,395] ({pool-2-thread-9} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1468364375373 finished by scheduler org.apache.zeppelin.livy.LivySparkRInterpreter1028865681
 INFO [2016-07-12 22:59:37,620] ({pool-2-thread-5} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1468364377620 started by scheduler org.apache.zeppelin.livy.LivySparkRInterpreter1028865681
ERROR [2016-07-12 22:59:37,633] ({pool-2-thread-5} LivySparkRInterpreter.java[interpret]:81) - Exception in LivySparkRInterpreter while interpret
org.springframework.web.client.HttpClientErrorException: 404 Not Found
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
        at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:354)
        at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:299)
        at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:240)
        at org.apache.zeppelin.livy.LivySparkRInterpreter.interpret(LivySparkRInterpreter.java:79)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
 INFO [2016-07-12 22:59:37,637] ({pool-2-thread-5} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1468364377620 finished by scheduler org.apache.zeppelin.livy.LivySparkRInterpreter1028865681

This issue looks like Livy interpreter specific.
The interpreter works fine after restarting livy interpreter.

Issue Links

• links to: GitHub Pull Request #1184