Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26422

Unable to disable Hive support in SparkR when Hadoop version is unsupported

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.3.3, 2.4.1, 3.0.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      When we make a Spark session as below:

      sparkSession <- sparkR.session("local[4]", "SparkR", Sys.getenv("SPARK_HOME"),
                                     list(spark.driver.extraClassPath = jarpaths,
                                          spark.executor.extraClassPath = jarpaths),
                                     enableHiveSupport = FALSE)
      

      I faced an issue that it's unable to disable Hive support explicitly with the error below:

      java.lang.reflect.InvocationTargetException
      ...
      Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1.3.1.0.0-78
      	at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
      	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
      	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
      	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
      	... 43 more
      Error in handleErrors(returnStatus, conn) :
        java.lang.ExceptionInInitializerError
      	at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:348)
      	at org.apache.spark.util.Utils$.classForName(Utils.scala:193)
      	at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1116)
      	at org.apache.spark.sql.api.r.SQLUtils$.getOrCreateSparkSession(SQLUtils.scala:52)
      	at org.apache.spark.sql.api.r.SQLUtils.getOrCreateSparkSession(SQLUtils.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:167)
      	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:108)
      ...
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: