Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
-
CentOS 6.7
Hadoop 2.6.1
HBase 1.1.2
PIG 0.15.0
JDK 1.7.0_21
Description
I am using Hadoop version 2.6.1, HBase version 1.1.2 and PIG version 0.15.0
I have also set the required environment variables in /home/hadoop/.bashrc file as follows
========================
- .bashrc
- Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
- User specific aliases and functions
export HADOOP_HOME=/opt/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
-
-
- Flume Home Directory
export FLUME_HOME=/usr/lib/flume/apache-flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASSPATH=$FLUME_CONF_DIR
export PATH=$PATH:$FLUME_HOME/bin
- Flume Home Directory
-
-
-
- HBase Home Directory
-
export HBASE_HOME=/usr/lib/hbase
export HBASE_CONF_DIR=$HBASE_HOME/conf
export HBASE_CLASSPATH=$HBASE_CONF_DIR
export PATH=$PATH:$HBASE_HOME/bin
###Hive Home Directory
export HIVE_HOME=/usr/lib/hive/apache-hive
export HIVE_CONF_DIR=$HIVE_HOME/conf
export HIVE_CLASSPATH=$HIVE_CONF_DIR
export PATH=$PATH:$HIVE_HOME/bin
###HCatalog Home Directory
export HCAT_HOME=$HIVE_HOME/hcatalog
export HCAT_CONF_DIR=$HCAT_HOME/etc/hcatalog:$HCAT_HOME/etc/webhcat
export HCAT_CLASSPATH=$HCAT_CONF_DIR
export PATH=$PATH:$HCAT_HOME/bin:$HCAT_HOME/sbin
###PIG Home Directory
export PIG_HOME=/usr/lib/pig/pig-0.15.0
export PIG_CONF_DIR=$PIG_HOME/conf
export PIG_CLASSPATH=$PIG_CONF_DIR:$PIG_HOME/pig-0.15.0-core-h2.jar:$HBASE_HOME
export PATH=$PATH:$PIG_HOME/bin
==========================
When I try to execute the following script
raw = LOAD 'hbase://test_table' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage ('test_cf:NAME') AS (NAME:chararray);
I get following error message
Pig Stack Trace
---------------
ERROR 1200: Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf:NAME]'
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf:NAME]'
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1748)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:631)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: Failed to parse: Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf:NAME]'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
... 15 more
Caused by:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf:NAME]'
at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897)
at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
... 16 more
Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf:NAME]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772)
at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885)
... 22 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
... 23 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:427)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:368)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:239)
... 28 more
================================================================================
The hbase-client-1.1.2.jar is in path of PIG which contains the org.apache.hadoop.hbase.client.Scan class.
Can someone please help if I am missing the configuration somewhere? Am I using the compatible versions of PIG and HBase.
I also tried all the suggestion found on internet to fix this problem, including compiling PIG, copying Jars, Registering Jars with PIG, etc. Nothing is working for me.
Thanks
Attachments
Issue Links
- relates to
-
PIG-4728 Compilation against hbase 1.x fails with hbase-hadoop1-compat not found
- Closed