Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-380

Livy should overwrite spark.sql.catalogImplementation if Hive classes are not present on classpath

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3
    • Fix Version/s: 0.4.0
    • Component/s: REPL
    • Labels:
      None

      Description

      In SparkContextInitializer.spark2CreateContext, livy checks for whether catalog implementation is "hive" or "in-memory" to properly initialize the SparkSession.
      If the config is set to "hive" (this is the case if livy.repl.enable-hive-context is set to "true"), livy will further check if hive classes are present or not.
      It only invokes the enableHiveSupport method of SparkSession.Builder if hive classes are found.
      If not, it proceeds to initializing a SparkSession without Hive support enabled.
      However, in Spark's code base, when creating SparkSession, it checks for spark.sql.catalogImplementation to determine which session state to initialize.
      If spark.sql.catalogImplementation is set to "hive", it will initialize a HiveSessionState.
      However, since Livy has already checked that hive classes are not present, it should not go ahead invoking SparkSession.builder.getOrCreate without overwriting spark.sql.catalogImplementation to "in-memory".

        Attachments

          Activity

            People

            • Assignee:
              mshen Min Shen
              Reporter:
              mshen Min Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: