Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1909

spark session with hive support can't be created correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: Interpreters
    • Labels:
      None

      Description

          if (useHiveContext()) {
            if (hiveClassesArePresent()) {
              Utils.invokeMethod(builder, "enableHiveSupport");
              sparkSession = Utils.invokeMethod(builder, "getOrCreate");
              logger.info("Created Spark session with Hive support");
            } else {
              Utils.invokeMethod(builder, "config",
                  new Class[]{ String.class, String.class},
                  new Object[]{ "spark.sql.catalogImplementation", "in-memory"});
              sparkSession = Utils.invokeMethod(builder, "getOrCreate");
              logger.info("Created Spark session with Hive support");
            }
          } 
      
       /**
         * See org.apache.spark.sql.SparkSession.hiveClassesArePresent
         * @return
         */
        private boolean hiveClassesArePresent() {
          try {
            this.getClass().forName("org.apache.spark.sql.hive.HiveSessionState");
            this.getClass().forName("org.apache.spark.sql.hive.HiveSharedState");
            this.getClass().forName("org.apache.hadoop.hive.conf.HiveConf");
            return true;
          } catch (ClassNotFoundException | NoClassDefFoundError e) {
            return false;
          }
        }
      

      since "org.apache.spark.sql.hive.HiveSharedState" has been removed from spark2.1.x

      So hiveClassesArePresent will always return false
      and spark session with hive support will always be created with
      "spark.sql.catalogImplementation=in-memory"

      Maybe just remove

            this.getClass().forName("org.apache.spark.sql.hive.HiveSharedState");
      

      is OK?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                licl lichenglin
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: