Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2706

Enable Spark to support Hive 0.13

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Dependency upgrade
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.1
    • 1.2.0
    • SQL
    • None

    Description

      It seems Spark cannot work with Hive 0.13 well.
      When I compiled Spark with Hive 0.13.1, I got some error messages, as attached below.
      So, when can Spark be enabled to support Hive 0.13?

      Compiling Error:

      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala:180: type mismatch;
      found : String
      required: Array[String]
      [ERROR] val proc: CommandProcessor = CommandProcessorFactory.get(tokens(0), hiveconf)
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:264: overloaded method constructor TableDesc with alternatives:
      (x$1: Class[_ <: org.apache.hadoop.mapred.InputFormat[_, _]],x$2: Class[_],x$3: java.util.Properties)org.apache.hadoop.hive.ql.plan.TableDesc <and>
      ()org.apache.hadoop.hive.ql.plan.TableDesc
      cannot be applied to (Class[org.apache.hadoop.hive.serde2.Deserializer], Class[(some other)?0(in value tableDesc)(in value tableDesc)], Class[?0(in value tableDesc)(in value tableDesc)], java.util.Properties)
      [ERROR] val tableDesc = new TableDesc(
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala:140: value getPartitionPath is not a member of org.apache.hadoop.hive.ql.metadata.Partition
      [ERROR] val partPath = partition.getPartitionPath
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScan.scala:132: value appendReadColumnNames is not a member of object org.apache.hadoop.hive.serde2.ColumnProjectionUtils
      [ERROR] ColumnProjectionUtils.appendReadColumnNames(hiveConf, attributes.map(_.name))
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:79: org.apache.hadoop.hive.common.type.HiveDecimal does not have a constructor
      [ERROR] new HiveDecimal(bd.underlying())
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:132: type mismatch;
      found : org.apache.hadoop.fs.Path
      required: String
      [ERROR] SparkHiveHadoopWriter.createPathFromString(fileSinkConf.getDirName, conf))
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:179: value getExternalTmpFileURI is not a member of org.apache.hadoop.hive.ql.Context
      [ERROR] val tmpLocation = hiveContext.getExternalTmpFileURI(tableLocation)
      [ERROR] ^
      [ERROR] /ws/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala:209: org.apache.hadoop.hive.common.type.HiveDecimal does not have a constructor
      [ERROR] case bd: BigDecimal => new HiveDecimal(bd.underlying())
      [ERROR] ^
      [ERROR] 8 errors found
      [DEBUG] Compilation failed (CompilerInterface)
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO]
      [INFO] Spark Project Parent POM .......................... SUCCESS [2.579s]
      [INFO] Spark Project Core ................................ SUCCESS [2:39.805s]
      [INFO] Spark Project Bagel ............................... SUCCESS [21.148s]
      [INFO] Spark Project GraphX .............................. SUCCESS [59.950s]
      [INFO] Spark Project ML Library .......................... SUCCESS [1:08.771s]
      [INFO] Spark Project Streaming ........................... SUCCESS [1:17.759s]
      [INFO] Spark Project Tools ............................... SUCCESS [15.405s]
      [INFO] Spark Project Catalyst ............................ SUCCESS [1:17.405s]
      [INFO] Spark Project SQL ................................. SUCCESS [1:11.094s]
      [INFO] Spark Project Hive ................................ FAILURE [11.121s]
      [INFO] Spark Project REPL ................................ SKIPPED
      [INFO] Spark Project YARN Parent POM ..................... SKIPPED
      [INFO] Spark Project YARN Stable API ..................... SKIPPED
      [INFO] Spark Project Assembly ............................ SKIPPED
      [INFO] Spark Project External Twitter .................... SKIPPED
      [INFO] Spark Project External Kafka ...................... SKIPPED
      [INFO] Spark Project External Flume ...................... SKIPPED
      [INFO] Spark Project External ZeroMQ ..................... SKIPPED
      [INFO] Spark Project External MQTT ....................... SKIPPED
      [INFO] Spark Project Examples ............................ SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 9:25.609s
      [INFO] Finished at: Wed Jul 23 05:22:06 EDT 2014
      [INFO] Final Memory: 52M/873M
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile (scala-compile-first) on project spark-hive_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile failed. CompileFailed -> [Help 1]
      org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile (scala-compile-first) on project spark-hive_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile failed.
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
      at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
      at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
      at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
      at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
      Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile failed.
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
      ... 19 more
      Caused by: Compilation failed
      at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76)
      at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35)
      at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29)
      at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71)
      at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71)
      at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71)
      at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101)
      at sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70)
      at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88)
      at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60)
      at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24)
      at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22)
      at sbt.inc.Incremental$.cycle(Incremental.scala:52)
      at sbt.inc.Incremental$.compile(Incremental.scala:29)
      at sbt.inc.IncrementalCompile$.apply(Compile.scala:20)
      at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:96)
      at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:44)
      at com.typesafe.zinc.Compiler.compile(Compiler.scala:158)
      at com.typesafe.zinc.Compiler.compile(Compiler.scala:142)
      at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:77)
      at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:308)
      at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
      at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:104)
      at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
      ... 20 more
      [ERROR]

      Attachments

        1. v1.0.2.diff
          62 kB
          Zhan Zhang
        2. spark-hive.err
          196 kB
          Ted Yu
        3. spark-2706-v2.txt
          12 kB
          Ted Yu
        4. spark-2706-v1.txt
          2 kB
          Ted Yu
        5. hive.diff
          12 kB
          Zhan Zhang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zhazhan Zhan Zhang
            chunjun.xiao Chunjun Xiao
            Votes:
            3 Vote for this issue
            Watchers:
            23 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment