Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.5.0, 0.5.1
-
None
Description
In Spark 2.3 cluster, use Zeppelin with livy2 interpreter, and type:
%sparkr sparkR.session()
You will see error:
[1] "Error in writeJobj(con, object): invalid jobj 1"
In a successful case with older livy and spark versions, we see something like this:
Java ref type org.apache.spark.sql.SparkSession id 1
This indicates isValidJobj() function in Spark code returned false for SparkSession obj. This is isValidJobj() function in Spark 2.3 code FYI:
isValidJobj <- function(jobj) { if (exists(".scStartTime", envir = .sparkREnv)) { jobj$appId == get(".scStartTime", envir = .sparkREnv) } else { FALSE } }