Hive
  1. Hive
  2. HIVE-6346

Add Hadoop-2.4.0 shims to hive-tez

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: Shims
    • Labels:
      None
    • Release Note:
      Add ZeroCopy and DirectDecompressor shims to make use of the HDFS zero-copy reads.

      Description

      The HadoopShims needs new shim functions to add extra HDFS Caching functionality which is not available in 2.2.0 branch.

      1. HIVE-6346.1.patch
        13 kB
        Gopal V
      2. HIVE-6346.2.patch
        24 kB
        Gopal V
      3. HIVE-6346.3.patch
        15 kB
        Gopal V
      4. HIVE-6346.4.patch
        15 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Gopal V added a comment -

          Patch to add 0.23C shims to hive-shims & hive-exec.

          Applies to hive/tez branch & pulls apache-snapshots to build against 2.4.0-SNAPSHOT release.

          Show
          Gopal V added a comment - Patch to add 0.23C shims to hive-shims & hive-exec. Applies to hive/tez branch & pulls apache-snapshots to build against 2.4.0-SNAPSHOT release.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626445/HIVE-6346.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 4993 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1147/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1147/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12626445

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626445/HIVE-6346.2.patch ERROR: -1 due to 1 failed/errored test(s), 4993 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1147/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1147/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12626445
          Hide
          Gunther Hagleitner added a comment -

          Gopal VComment on rb. Looks good so far. Biggest question I have is whether we need another shim version for that. Seems we could just upgrade 23 when 2.4 is out.

          Also: The test failure is unrelated. Tests have been successful.

          Show
          Gunther Hagleitner added a comment - Gopal V Comment on rb. Looks good so far. Biggest question I have is whether we need another shim version for that. Seems we could just upgrade 23 when 2.4 is out. Also: The test failure is unrelated. Tests have been successful.
          Hide
          Gopal V added a comment -

          Removed new Shim and rolled the feature-set into a package protected class.

          That class is loaded only if the HDFS features are available during shim load.

          Show
          Gopal V added a comment - Removed new Shim and rolled the feature-set into a package protected class. That class is loaded only if the HDFS features are available during shim load.
          Hide
          Gunther Hagleitner added a comment -

          Looks good to me. Another couple of nits on rb. I can change those when I commit (if that's fine with you).

          Show
          Gunther Hagleitner added a comment - Looks good to me. Another couple of nits on rb. I can change those when I commit (if that's fine with you).
          Hide
          Gopal V added a comment -

          Sure.

          I just did some testing with hadooo-2.2.x release with zero-copy reads config option turned on, with a build compiled against 2.4.0. The feature gets turned off automatically without any class loader errors.

          Show
          Gopal V added a comment - Sure. I just did some testing with hadooo-2.2.x release with zero-copy reads config option turned on, with a build compiled against 2.4.0. The feature gets turned off automatically without any class loader errors.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626834/HIVE-6346.3.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1180/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1180/console

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.20 ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.20 ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.20 ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.20 ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.20 ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/hive-shims-0.20-0.13.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20 ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/hive-shims-0.20-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20/0.13.0-SNAPSHOT/hive-shims-0.20-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20/0.13.0-SNAPSHOT/hive-shims-0.20-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims Secure Common 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-common-secure ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-common-secure ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-common-secure ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-common-secure ---
          [INFO] Compiling 12 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/classes
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java uses or overrides a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [WARNING] Note: Some input files use unchecked or unsafe operations.
          [WARNING] Note: Recompile with -Xlint:unchecked for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-common-secure ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-common-secure ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-common-secure ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-common-secure ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-common-secure ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/hive-shims-common-secure-0.13.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-common-secure ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/hive-shims-common-secure-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-common-secure/0.13.0-SNAPSHOT/hive-shims-common-secure-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-common-secure/0.13.0-SNAPSHOT/hive-shims-common-secure-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims 0.20S 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.20S ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.20S ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.20S ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.20S ---
          [INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/classes
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java uses or overrides a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.20S ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.20S ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.20S ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.20S ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.20S ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-yarn-api:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-yarn-common:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-yarn-client:jar:2.3.0-SNAPSHOT is missing, no dependency information available
          [WARNING] The POM for org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available
          Downloading: http://www.datanucleus.org/downloads/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar
          Downloading: http://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar
          Downloaded: http://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar (973 KB at 483.9 KB/sec)
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [4.722s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [8.607s]
          [INFO] Hive Shims Common ................................. SUCCESS [2.649s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.161s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [2.985s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [1.527s]
          [INFO] Hive Shims 0.23 ................................... FAILURE [5.301s]
          [INFO] Hive Shims ........................................ SKIPPED
          [INFO] Hive Common ....................................... SKIPPED
          [INFO] Hive Serde ........................................ SKIPPED
          [INFO] Hive Metastore .................................... SKIPPED
          [INFO] Hive Query Language ............................... SKIPPED
          [INFO] Hive Service ...................................... SKIPPED
          [INFO] Hive JDBC ......................................... SKIPPED
          [INFO] Hive Beeline ...................................... SKIPPED
          [INFO] Hive CLI .......................................... SKIPPED
          [INFO] Hive Contrib ...................................... SKIPPED
          [INFO] Hive HBase Handler ................................ SKIPPED
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 31.032s
          [INFO] Finished at: Tue Feb 04 11:34:09 EST 2014
          [INFO] Final Memory: 24M/59M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal on project hive-shims-0.23: Could not resolve dependencies for project org.apache.hive.shims:hive-shims-0.23:jar:0.13.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-hdfs:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-hdfs:jar:tests:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:tests:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-api:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-common:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-client:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.3.0-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-shims-0.23
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12626834

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626834/HIVE-6346.3.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1180/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1180/console Messages: **** This message was trimmed, see log for full details **** [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.20 --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.20 --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.20 --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.20 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.20 --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/hive-shims-0.20-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20 --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/target/hive-shims-0.20-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20/0.13.0-SNAPSHOT/hive-shims-0.20-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20/0.13.0-SNAPSHOT/hive-shims-0.20-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims Secure Common 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-common-secure --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-common-secure --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-common-secure --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-common-secure --- [INFO] Compiling 12 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java uses or overrides a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-common-secure --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-common-secure --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-common-secure --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-common-secure --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-common-secure --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/hive-shims-common-secure-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-common-secure --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/target/hive-shims-common-secure-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-common-secure/0.13.0-SNAPSHOT/hive-shims-common-secure-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/common-secure/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-common-secure/0.13.0-SNAPSHOT/hive-shims-common-secure-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.20S 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.20S --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.20S --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.20S --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.20S --- [INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java uses or overrides a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.20S --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.20S --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.20S --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.20S --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.20S --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-yarn-api:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-yarn-common:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-yarn-client:jar:2.3.0-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.3.0-SNAPSHOT is missing, no dependency information available Downloading: http://www.datanucleus.org/downloads/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar Downloading: http://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar Downloaded: http://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.jar (973 KB at 483.9 KB/sec) [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [4.722s] [INFO] Hive Ant Utilities ................................ SUCCESS [8.607s] [INFO] Hive Shims Common ................................. SUCCESS [2.649s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.161s] [INFO] Hive Shims Secure Common .......................... SUCCESS [2.985s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.527s] [INFO] Hive Shims 0.23 ................................... FAILURE [5.301s] [INFO] Hive Shims ........................................ SKIPPED [INFO] Hive Common ....................................... SKIPPED [INFO] Hive Serde ........................................ SKIPPED [INFO] Hive Metastore .................................... SKIPPED [INFO] Hive Query Language ............................... SKIPPED [INFO] Hive Service ...................................... SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 31.032s [INFO] Finished at: Tue Feb 04 11:34:09 EST 2014 [INFO] Final Memory: 24M/59M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hive-shims-0.23: Could not resolve dependencies for project org.apache.hive.shims:hive-shims-0.23:jar:0.13.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-hdfs:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-hdfs:jar:tests:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:tests:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-api:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-common:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-client:jar:2.3.0-SNAPSHOT, org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.3.0-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-common:jar:2.3.0-SNAPSHOT -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-shims-0.23 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12626834
          Hide
          Gunther Hagleitner added a comment -

          Committed to branch.

          Show
          Gunther Hagleitner added a comment - Committed to branch.

            People

            • Assignee:
              Gopal V
              Reporter:
              Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development