Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16531 Enable support for HADOOP-3.x
  3. HIVE-16532

HIVE on hadoop 3 build failed due to hdfs client/server jar separation

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      Just like TEZ-3690, hdfs-client jar separated from hadoop-hdfs will cause build failures, something like:

      ...
07:00:01 2017/04/25 14:00:00 INFO    : [INFO] Executed tasks
07:00:01 2017/04/25 14:00:00 INFO    : [INFO]
07:00:01 2017/04/25 14:00:00 INFO    : [INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ hive-shims-0.23 ---
07:00:01 2017/04/25 14:00:00 INFO    : [INFO] Compiling 4 source files to /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/target/classes
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API.
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: Recompile with -Xlint:deprecation for details.
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses unchecked or unsafe operations.
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: Recompile with -Xlint:unchecked for details.
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] -------------------------------------------------------------
07:00:02 2017/04/25 14:00:01 INFO    : [ERROR] COMPILATION ERROR :
07:00:02 2017/04/25 14:00:01 INFO    : [INFO] -------------------------------------------------------------
…
07:00:02 2017/04/25 14:00:02 INFO    : [WARNING] The requested profile "hadoop-2" could not be activated because it does not exist.
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hive-shims-0.23: Compilation failure: Compilation failure:
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[59,30] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DFSClient
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[61,30] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DistributedFileSystem
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[64,37] package org.apache.hadoop.hdfs.client does not exist
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[65,39] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DirectoryListing
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs.protocol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[66,39] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class EncryptionZone
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs.protocol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[67,39] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class HdfsConstants
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs.protocol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[68,39] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class HdfsLocatedFileStatus
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: package org.apache.hadoop.hdfs.protocol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[580,38] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DFSClient
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[773,11] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DistributedFileSystem
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1045,19] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DistributedFileSystem
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.StoragePolicyShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1047,30] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DistributedFileSystem
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.StoragePolicyShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1140,13] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class HdfsAdmin
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.HdfsEncryptionShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1158,41] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class DFSClient
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.HdfsEncryptionShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1195,47] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class EncryptionZone
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.HdfsEncryptionShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1195,69] cannot find symbol
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] symbol:   class EncryptionZone
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] location: class org.apache.hadoop.hive.shims.Hadoop23Shims.HdfsEncryptionShim
07:00:02 2017/04/25 14:00:02 INFO    : [ERROR] /grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[570,23] cannot access org.apache.hadoop.hdfs.DistributedFileSystem
...

      Attachments

        Issue Links

          Is contained by

          Sub-task - The sub-task of the issue HIVE-15016 Run tests with Hadoop 3.0.0-beta1

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              junping_du Junping Du
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: