Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26422

Unable to disable Hive support in SparkR when Hadoop version is unsupported

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 2.3.3, 2.4.1, 3.0.0
    • SparkR
    • None

    Description

      When we make a Spark session as below:

      sparkSession <- sparkR.session("local[4]", "SparkR", Sys.getenv("SPARK_HOME"),
                                     list(spark.driver.extraClassPath = jarpaths,
                                          spark.executor.extraClassPath = jarpaths),
                                     enableHiveSupport = FALSE)
      

      I faced an issue that it's unable to disable Hive support explicitly with the error below:

      java.lang.reflect.InvocationTargetException
      ...
      Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1.3.1.0.0-78
      	at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
      	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
      	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
      	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
      	... 43 more
      Error in handleErrors(returnStatus, conn) :
        java.lang.ExceptionInInitializerError
      	at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:348)
      	at org.apache.spark.util.Utils$.classForName(Utils.scala:193)
      	at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1116)
      	at org.apache.spark.sql.api.r.SQLUtils$.getOrCreateSparkSession(SQLUtils.scala:52)
      	at org.apache.spark.sql.api.r.SQLUtils.getOrCreateSparkSession(SQLUtils.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:167)
      	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:108)
      ...
      

      Attachments

        Issue Links

          Activity

            People

              hyukjin.kwon Hyukjin Kwon
              hyukjin.kwon Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: