[LIVY-505] sparkR.session failed with "invalid jobj 1" error in Spark 2.3

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.5.0, 0.5.1
• Fix Version/s: 0.9.0
• Component/s: Interpreter
• Assignee: Unassigned
• Reporter: shanyu zhao
• Created: 29/Aug/18 02:10
• Updated: 12/Nov/22 22:09

Description

In Spark 2.3 cluster, use Zeppelin with livy2 interpreter, and type:

%sparkr
sparkR.session()

You will see error:

[1] "Error in writeJobj(con, object): invalid jobj 1"

In a successful case with older livy and spark versions, we see something like this:

Java ref type org.apache.spark.sql.SparkSession id 1

This indicates isValidJobj() function in Spark code returned false for SparkSession obj. This is isValidJobj() function in Spark 2.3 code FYI:

isValidJobj <- function(jobj) {
  if (exists(".scStartTime", envir = .sparkREnv)) {
    jobj$appId == get(".scStartTime", envir = .sparkREnv)
  } else {
    FALSE
  }
}