Details
-
Wish
-
Status: Resolved
-
Minor
-
Resolution: Won't Fix
-
0.12.0
-
None
-
Hadoop 2.2 local derby Meatastore embedded
-
hive hadoop 2.2 datanucleus
Description
There is an with hive 12 datanucleus incompatability which seems to have invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
The main question:
IF hive 0.12.0 and datanucleus are compatabile, then what is the version of datanucleus I should be using with Hive 12 and Hadoop 2.2?
The error which Im getting (this blocks me from properly running hive queries invoked from the "test" phase of a maven project)
To reproduce
I have hadoop and hive running as a pseudo cluster local mode and derby as the metastore
I have the following environment variables
HADOOP_HOME=/home/ubu/hadoop JAVA_HOME=/usr/lib/jvm/java-7-oracle
I have the RegexSerDe declared in the hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
<description>This JAR file available to all users for alljobs</description>
</property>
If I run with
<datanucleus.version>3.0.2</datanucleus.version>
I get the following 1 exception only
'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
HOWEVER, If I run with
<datanucleus.version>3.2.0-release</datanucleus.version>
I get the following 1 exception exception only
java.lang.ClassNotFoundException:
org/apache/hadoop/hive/contrib/serde2/RegexSerDe
EXPLANATION
The RegexSerDe class is picked up at run time but the datanucleus Map class is not available, I have checked in the datanucleus-core 3.0.2 jar and it is missing, Upgrading to the first datanucleus above 3.0.2 that includes the Map class throws the ClassNotFoundException for RegexSerDe.
The earlier 3.0.2 datanucleus, code fails with the missing Map class but the RegexSerDe class is found, then when I upgrade to the
3.2.0-release the Map class is found but for some unkown reason the code/Hive no longer finds the RegexSerDe class
I started using the same datanucleus dependencies found in this hive pom
http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
below are the dependencies my latest attempts to get a functioning pom
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<!-- misc -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>${derby.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>${datanucleus.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
<version>${datanucleus-rdbms.version}</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>${datanucleus.jdo.version}</version>
<exclusions>
<exclusion>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
</exclusion>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- hadoop -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- hive -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-common</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-serde</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-contrib</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-cli</artifactId>
<version>${hive.version}</version>
<exclusions>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.jolbox</groupId>
<artifactId>bonecp</artifactId>
<version>${bonecp.version}</version>
</dependency>
<!-- logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- SL4J Binding provided at runtime -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.12</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Unit test artifacts -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.0.0</version>
<classifier>hadoop2</classifier>
</dependency>