Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-505

sparkR.session failed with "invalid jobj 1" error in Spark 2.3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.5.0, 0.5.1
    • Fix Version/s: None
    • Component/s: Interpreter
    • Labels:
      None

      Description

      In Spark 2.3 cluster, use Zeppelin with livy2 interpreter, and type:

      %sparkr
      sparkR.session()

      You will see error:

      [1] "Error in writeJobj(con, object): invalid jobj 1"

      In a successful case with older livy and spark versions, we see something like this:

      Java ref type org.apache.spark.sql.SparkSession id 1

      This indicates isValidJobj() function in Spark code returned false for SparkSession obj. This is isValidJobj() function in Spark 2.3 code FYI:

      isValidJobj <- function(jobj) {
        if (exists(".scStartTime", envir = .sparkREnv)) {
          jobj$appId == get(".scStartTime", envir = .sparkREnv)
        } else {
          FALSE
        }
      }

        Attachments

        1. LIVY-505.patch
          0.9 kB
          shanyu zhao

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shanyu shanyu zhao
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: