Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-8758

Fix hadoop-1 build [Spark Branch]

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark
    • None


      This may mean merging patches from trunk and fixing whatever problem specific to Spark branch. Here are user reported problems:

      Problem 1:

      Hive Serde ......................................... FAILURE [  2.357 s]
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-serde: Compilation failure: Compilation failure:
      [ERROR] /data/hive-spark/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:[27,24] cannot find symbol
      [ERROR] symbol:   class Nullable
      [ERROR] location: package javax.annotation
      [ERROR] /data/hive-spark/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:[67,36] cannot find symbol
      [ERROR] symbol:   class Nullable
      [ERROR] location: class org.apache.hadoop.hive.serde2.AbstractSerDe

      My understanding: Looks the Nullable annotation was recently added in the recent branch. Added the below dependency in the project hive-serde


      Problem 2:
      After adding the dependency for hive-serde, got the below compilation error

      [INFO] Hive Query Language ................................ FAILURE [01:35 min]
      /data/hive-spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java:[35,39] error: package org.apache.hadoop.mapreduce.util does not exist

      In the dependency jar for hadoop-1 (hadoop-core-1.2.1.jar) - We do not have the package “org.apache.hadoop.mapreduce.util” to circumvent it added the below dependency where we had the package (not sure, it is right – I badly wanted to make the build successful L)


      Problem 3:
      After making the above change, again failed in the same project @ file /data/hive-spark/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java. In the snippet below taken from the file, we can see the “fileStatus.isFile()” is called which is not available in the “org.apache.hadoop.fs.FileStatus” hadoop1 api.

       for (FileStatus fileStatus: fs.listStatus(folder)) {
             Path filePath = fileStatus.getPath();
              if (!fileStatus.isFile()) {
                throw new HiveException("Error, not a file: " + filePath);


        1. HIVE-8758.1-spark.patch
          6 kB
          Jimmy Xiang
        2. HIVE-8758.2-spark.patch
          9 kB
          Jimmy Xiang

        Issue Links


          This comment will be Viewable by All Users Viewable by All Users


            jxiang Jimmy Xiang Assign to me
            xuefuz Xuefu Zhang
            0 Vote for this issue
            2 Start watching this issue




                Issue deployment