Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.12.0, 0.13.0
-
Fix Version/s: 0.14.0
-
Component/s: HBase Handler
-
Labels:None
Description
Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12626065/HIVE-6147.1.patch.txt
ERROR: -1 due to 22 failed/errored test(s), 4987 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed
This message is automatically generated.
ATTACHMENT ID: 12626065
This looks good, but from the patch, it seems that the solution is only for HBase. I wonder if we have given thoughts on the idea of generalizing the problem and providing a general solution. I can see the benefits of separating the storage (such as hbase) and data format (avro, thrift, protocol buf, parquet, etc). Then we solve M + N problems rather than M * N problems. What if the avro data is coming from other storage, such as accumulo, or parquet data from HBase.
Attaching new patch with very minor formatting updates and adding missing license headers.
Xuefu Zhang You are correct. This adds support to the existing HBaseSerDe via ObjectInspectors to query avro data stored in HBase. Also you make a very good point about having a unified interface to allow for the querying of any structured format stored in any storage like HBase or HDFS. Specifically we can look at creating specific implementations of the HBase DataType[1] to have such layer on top of HBase. The idea here though this specific problem and get this support in for now and then move towards a more generalized approach.
[1] https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/types/DataType.html
Review Request: https://reviews.apache.org/r/17566/
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12626225/HIVE-6147.2.patch.txt
ERROR: -1 due to 21 failed/errored test(s), 4987 tests executed
Failed tests:
org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed
This message is automatically generated.
ATTACHMENT ID: 12626225
Brock Noland,Xuefu Zhang If one of you get a chance to do a quick review of this, I would really appreciate that.
Thanks,
Swarnim Kulkarni Thanks for submitting the patch. Have you investigated the test failures shown above?
Yeah. I did dig into the test report and the failures seem unrelated to the code that this patch touches.
I have rebased the branch to master to see if it fixes anything. If these tests still fail, I'll start digging deeper.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
ERROR: -1 due to 24 failed/errored test(s), 5140 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed
This message is automatically generated.
ATTACHMENT ID: 12629556
I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed!
mac-swarnim:hive swarnim$ git pull --rebase
Current branch trunk is up to date.
mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
--2014-02-23 18:40:27-- https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
Resolving issues.apache.org... 140.211.11.121
Connecting to issues.apache.org|140.211.11.121|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 212743 (208K) [text/plain]
Saving to: `HIVE-6147.3.patch.txt'
100%[=====================================================================================================================>] 212,743 29.5K/s in 7.0s
2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743]
mac-swarnim:hive swarnim$ git status
# On branch trunk
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# HIVE-6147.3.patch.txt
nothing added to commit but untracked files present (use "git add" to track)
mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt
patching file hbase-handler/pom.xml
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java
patching file hbase-handler/src/if/avro/avro_test.avpr
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
mac-swarnim:hive swarnim$ git status
# On branch trunk
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: hbase-handler/pom.xml
# modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
# modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
# modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
# modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
# modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
# modified: serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# HIVE-6147.3.patch.txt
# hbase-handler/src/gen/
# hbase-handler/src/if/
# hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
# hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
# serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
# serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
# serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
mac-swarnim:hive swarnim$ cd hcatalog/
mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter ---
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = [])
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = [])
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter ---
[INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter ---
[INFO] Executing tasks
main:
[mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp
[mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse
[mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
[copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter ---
[INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter ---
[INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hcatalog.pig.TestHCatLoader
2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader
Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
Running org.apache.hcatalog.pig.TestHCatLoaderStorer
2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer
Running org.apache.hcatalog.pig.TestHCatStorer
2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer
Running org.apache.hcatalog.pig.TestHCatStorerMulti
2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti
Running org.apache.hcatalog.pig.TestHCatStorerWrapper
2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper
Running org.apache.hcatalog.pig.TestOrcHCatLoader
2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader
Running org.apache.hcatalog.pig.TestOrcHCatStorer
2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer
Running org.apache.hcatalog.pig.TestPigHCatUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil
Running org.apache.hive.hcatalog.pig.TestE2EScenarios
2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios
Running org.apache.hive.hcatalog.pig.TestHCatLoader
2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader
Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorer
2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti
2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti
Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader
2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader
Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer
2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer
Running org.apache.hive.hcatalog.pig.TestPigHCatUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil
Results :
Tests run: 101, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter ---
[INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter ---
[INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
[INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9:11.629s
[INFO] Finished at: Sun Feb 23 18:51:54 CST 2014
[INFO] Final Memory: 24M/81M
[INFO] ------------------------------------------------------------------------
Reattaching patch to re-run the pre-commit tests on trunk to see if the results differ from my local run.
I was looking for this feature, as Hbase itself is hard to handle 1 to M relationship, or support nest structure data, if we are only using Hbase as a DW storage. If we can store the AVRO bytes in one column family of Hbase table, and export that Avro schema out in the Hive, that can give us random update/insert the data as AVRO in Hbase, and use it in Hive or MR jobs generated from Hive.
Xuefu Zhang Any way we can get the pre-commit tests running on this guy? Seems like with the latest attached patch, the test did not run.
Swarnim Kulkarni Right now the queue for test runs are long, and it may take a day or two before you see the result. I have just manually created a test job in the queue.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12630595/HIVE-6147.3.patch.txt
ERROR: -1 due to 25 failed/errored test(s), 5186 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed
This message is automatically generated.
ATTACHMENT ID: 12630595
To circle back on this one, I re-ran the tests locally again and they all seem to pass for me(See the screenshot above)
Swarnim Kulkarni I'm not totally convinced that these tests are unrelated, as they consistently appeared in the test result. In addition, I manually ran TestHCatLoader, and got errors as the following:
testProjectionsBasic(org.apache.hive.hcatalog.pig.TestHCatLoader) Time elapsed: 0.184 sec <<< ERROR! java.io.IOException: Failed to execute "create table junit_unparted_complex(name string, studentid int, contact struct<phno:string,email:string>, currently_registered_courses array<string>, current_grades map<string,string>, phnos array<struct<phno:string,type:string>>) stored as RCFILE tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver')". Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException at org.apache.hive.hcatalog.pig.TestHCatLoader.executeStatementOnDriver(TestHCatLoader.java:125) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:111) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:101) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:115) at org.apache.hive.hcatalog.pig.TestHCatLoader.setup(TestHCatLoader.java:154)
Please further investigate.
So I cleaned up my local m2 repository, rebased the master and re-applied the patch and saw the test failure. May be it was a bad dep sitting in my repo that was causing the problem.
Attaching a new patch. If the tests pass with this, I will update the RB with this latest patch.
Overall: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633416/HIVE-6147.4.patch.txt
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1672/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/parquet_timestamp.q.out ql/src/test/results/clientnegative/parquet_char.q.out ql/src/test/results/clientnegative/parquet_date.q.out ql/src/test/results/clientnegative/parquet_decimal.q.out ql/src/test/results/clientnegative/parquet_varchar.q.out ql/src/test/queries/clientnegative/parquet_char.q ql/src/test/queries/clientnegative/parquet_timestamp.q ql/src/test/queries/clientnegative/parquet_decimal.q ql/src/test/queries/clientnegative/parquet_date.q ql/src/test/queries/clientnegative/parquet_varchar.q
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1575684.
At revision 1575684.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
This message is automatically generated.
ATTACHMENT ID: 12633416
New patch rebased with master.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633602/HIVE-6147.5.patch.txt
ERROR: -1 due to 1 failed/errored test(s), 5381 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed
This message is automatically generated.
ATTACHMENT ID: 12633602
Xuefu Zhang As the previously failing tests now pass, I have updated the RB with the latest patch for review.
Thanks for working on this, Swarnim Kulkarni.
1. This patch might have an ineligible overlap with HIVE-6411. Does it make sense to get that in first? Please work with Nevis regarding that.
2. For the generated test code, please make sure that it will not make its way to the production jars.
3. Avoid trailing spaces/tabs.
4. Please reuse AvroSerde code rather than P&C.
5. Avoid catching individual exception even if the handling of the exception is the same.
Thanks Xuefu Zhang for reviewing. I agree it makes lot of sense for HIVE-6411 to go in first and then I can refactor this on the basis of that.
Also on the point of reusing AvroSerDe code, I have tried to write AvroLazyObjectInspector simply as a wrapper on top of AvroSerDe still delegating most of the operations to the serde. Any specific instance P&C instance you want me to look deeper into?
Swarnim Kulkarni I'm glad that you have the principle of code reuse in mind. I only browsed the patch, and spotted HiveSerdeHelper.getSchemaFromFS(), which is seemingly for the same purpose as AvroSerdeUtils.getSchemaFromFS() is. This might be coincidental. No big deal.
Swarnim Kulkarni very sorry this review has taken so long. Can you rebase the current patch? I also see there are many extra newlines in several methods like getTestAvroBytesFromClass2. Can you remove those as well?
I will review the updated patch promptly.
Swarnim Kulkarni do you have a few minutes to rebase this patch?
Thanks for your reply Brock Noland. Actually there is a little more cleanup work that is to be done with this patch as well along with rebasing. I should have a chance to get to this sometime this week. Thanks again.
Attaching refactored patch for testing.
Brock NolandXuefu Zhang Updated RB for the patch: https://reviews.apache.org/r/17566/
Swarnim Kulkarni for some reason I cannot change the JIRA to "Patch Available" so tests can run. Do you have the button?
Brock Noland Just did that.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667104/HIVE-6147.6.patch.txt
ERROR: -1 due to 3 failed/errored test(s), 6192 tests executed
Failed tests:
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/695/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/695/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-695/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed
This message is automatically generated.
ATTACHMENT ID: 12667104
+1
One thing to note here is that this doesn't support serializing of avro data into HBase yet. Should be pretty straightforward to add that in on top of this patch. Logged HIVE-8020 for that.
Thank you Swarnim! I have committed this to trunk!
Doc question: Will this be documented in the HBase Integration design doc or the Avro SerDe doc, or a new doc? (The HBase doc has a list of open issues, but this one isn't on the list.)
This has been fixed in 0.14 release. Please open new jira if you see any issues.
documentation on the usage of this that we can add to the Hive wiki
Swarnim Kulkarni, someone just asked me how to use this. Did you happen to have those docs, even if in a rough draft form?
Brock Noland Apologies for getting back to you so late. Some how I completely missed the notification on this.
Using this support is pretty straight forward. An example query looks like this:
CREATE EXTERNAL TABLE test_hbase_avro
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,test_col_fam:test_col","test_col_fam.test_col.serialization.type"="avro",test_col_fam.test_col.avro.schema.url="hdfs://testcluster/tmp/schema.avsc")
TBLPROPERTIES ("hbase.table.name" = "hbase_avro_table", "hbase.struct.autogenerate"="true");
So basically this looks exactly a query that we would use to query an hbase table. The only difference here being:
"test_col_fam.test_col.serialization.type"="avro"
Using this property, we are telling hive that the given column under the given column family is an avro column, so we need to deserialize it accordingly.
test_col_fam.test_col.avro.schema.url="hdfs://testcluster/tmp/schema.avsc"
Using this property you specify where is the reader schema for the column that will be used to deserialize. This can be on HDFS like mentioned here, or provided inline using something like "test_col_fam.test_col.avro.schema.literal" property. If you have a custom store where you store this schema, you can write a custom implementation of AvroSchemaRetriever[1] and plug that in using the "avro.schema.retriever property" using a property like "test_col_fam.test_col.avro.schema.retriever". Ofcourse you would need to ensure that the jar having this custom class is on the hive classpath.
"hbase.struct.autogenerate"="true"
Avro schemas can be complicated and deeply nested. So at times manually creating the columns and types for them is not feasible. Specifying this property lets hive auto deduce the columns and types using the schema that was provided.
Please do let me know if there are any more questions that I can help out with.
I'm trying to create a table in Hive 0.14 that points to an HBase table with one column family ("c") and one column ("b") that contains schema-less avro serialized object:
CREATE EXTERNAL TABLE customers ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( "hbase.columns.mapping" = ":key,c:b", "c.b.serialization.type"="avro", "c.b.avro.schema.url"="hdfs:/....../Customer.avsc") TBLPROPERTIES ("hbase.table.name" = "customers", "hbase.struct.autogenerate"="true", "hive.serialization.extend.nesting.levels"="true");
The DDL above creates the table successfully, but queries fail with the following error:
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating c_b 16/01/29 15:36:55 [main]: ERROR CliDriver: Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating c_b java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating c_b at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:152) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1621) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:267) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating c_b at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:82) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:571) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:563) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:138) ... 12 more Caused by: org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorException: An error occurred retrieving schema from bytes at org.apache.hadoop.hive.serde2.avro.AvroLazyObjectInspector.retrieveSchemaFromBytes(AvroLazyObjectInspector.java:331) at org.apache.hadoop.hive.serde2.avro.AvroLazyObjectInspector.deserializeStruct(AvroLazyObjectInspector.java:287) at org.apache.hadoop.hive.serde2.avro.AvroLazyObjectInspector.getStructFieldData(AvroLazyObjectInspector.java:142) at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:109) at org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldData(DelegatedStructObjectInspector.java:88) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:77) ... 17 more Caused by: java.io.IOException: Not a data file. at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105) at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:84) at org.apache.hadoop.hive.serde2.avro.AvroLazyObjectInspector.retrieveSchemaFromBytes(AvroLazyObjectInspector.java:328) ... 25 more
It seems that there is a problem in the following code in AvroLazyObjectInspector:
... private Object deserializeStruct(Object struct, String fieldName) { ... if (readerSchema == null) { ... } else { // a reader schema was provided if (schemaRetriever != null) { // a schema retriever has been provided as well. Attempt to read the write schema from the // retriever ws = schemaRetriever.retrieveWriterSchema(data); if (ws == null) { throw new IllegalStateException( "Null writer schema retrieved from schemaRetriever for field [" + fieldName + "]"); } } else { // attempt retrieving the schema from the data ws = retrieveSchemaFromBytes(data); } rs = readerSchema; try { avroWritable.readFields(data, ws, rs); } catch (IOException ioe) { throw new AvroObjectInspectorException("Error deserializing avro payload", ioe); } } ... } ...
because it tries to retrieve the write schema from data (ws = retrieveSchemaFromBytes(data)) even if the schema URL (reader schema) had been provided. Is there way to make it work for schema-less avro data?
it tries to retrieve the write schema from data (ws = retrieveSchemaFromBytes(data)) even if the schema URL (reader schema) had been provided
Correct. That is the default behavior. The writer schema defaults to the reader schema if one has not been provided. If it has been(like you are doing in your case), it would use the reader schema from the given URL but still default to the writer schema from the data. If you want to provide the writer schema as well, I would recommend you take a look into the AvroSchemaRetriever[1]. You can provide a custom implementation of it and provide both reader and write schema from any custom source that you would like. A test implementation can be found here for reference[2] and the corresponding test that uses this implementation here[3]. Once done, simply plug it in with "avro.schema.retriever" property. One caveat is that this will currently apply to the whole table and not individual columns. So it makes the assumption that there is a uniform schema across the table.
Hope this helps. Let me know if there are any additional questions.
[1] https://github.com/apache/hive/blob/release-1.2.1/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
[2] https://github.com/apache/hive/blob/release-1.2.1/hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
[3] https://github.com/apache/hive/blob/release-1.2.1/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java#L1293-L1344
Thank you for a quick reply, Swarnim. I'll try to implement a custom AvroSchemaRetriever. However, I'm not quite clear on why the writer schema is necessarily loaded if the reader schema is provided. It is pretty common to use schema-less avro objects in HBase (if there are billions of rows with objects of the same type, it is not reasonable to store the same schema in all of them) and it is not convenient to write a customer schema retriever for each such case. So, I wonder if it is better to assume that the writer schema is equal to the reader schema if the former cannot be found neither in data nor via a customer retriever?
It is pretty common to use schema-less avro objects in HBase.
I am not sure if that is true(if possible at all). As far as my understanding goes, you will have to almost always provide the exact schema that was used while persisting the data when attempting to deserialize it and the best way to do that would be to store alongside the schema itself. Plus schema evolution is going to be a mess. Imagine writing a billion rows in HBase with one schema which evolves and then you write another billion rows with new schema. How do you ensure the first billion rows are still correctly readable?
(if there are billions of rows with objects of the same type, it is not reasonable to store the same schema in all of them) and it is not convenient to write a customer schema retriever for each such case.
Correct. I agree it is inefficient to store it for every single cell. Although IMO that isn't a good excuse to not write the schema at all. A better design in this case is to use some kind of schema registry, use a custom serializer, write the schema to the schema registry, generate a id of some kind and persist the id along with the data. Then when you are reading the data, use the id to pull the schema from the store and read the data. That is also where a custom implementation of an AvroSchemaRetriever makes sense where your custom implementation would know how to read your schema from the schema registry and get that to hive and let hive handle the deserialization from there on.
> How do you ensure the first billion rows are still correctly readable?
Avro supports schema evolution that allows data to be written with one schema and read with another, as described, for example, here https://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/schemaevolution.html (section How Schema Evolution Works). Of course, feasibility of this feature depends on the application and how drastically the schema can be updated, but many applications can commit on backward compatible schema changes that make it appropriate to use the latest reader schema for data items including the old ones.
Avro supports schema evolution that allows data to be written with one schema and read with another
Yup. Definitely agree. However the point I was trying to make is that you would still need to provide the same exact schema that was used when writing the data. Let's take an example. Let's say you used Schema S1 to write a billion rows to HBase. The Schema then evolved to S2(hopefully in a compatible way) and you write another billion rows with it. The Schema evolves again to S3 and then you write another billion rows. Now to be able to read all this data, this is what you would need to do.
1st billion rows:
Writer Schema: S1
Reader Schema: S3
2nd billion rows:
Writer Schema: S2
Reader Schema: S3
3rd billion rows:
Writer Schema: S3
Reader Schema: S3
So as you see, you are still providing the exact same version of the schema that was used to write the data to be able to read it back successfully. Without it, it would be extremely hard for avro for make out head and tail of our data. You "might" still get lucky and be able to deserialize the 1st billion rows using S3 as reader/writer schema but there are absolutely no guarantees whatsoever. Which is why you would still need a way regardless to track what schema was used to write the persist the data when you read it back and the current design of hive/hbase avro support closely follows that pattern.
Thanks for writing the documentation, Swarnim Kulkarni. I have included what you wrote with some minor edits in the Hive wiki:
Please let me know if there is anything you would like changed or clarified (for example, I added quotation marks around the "test_col_fam.test_col.avro.schema.url" property for consistency, but if the lack of quotation marks originally was intentional, this can be changed).
Swarnim Kulkarni, could you please review the docs in the wiki?
If they're okay, you can remove the TODOC14 label from this issue. Thanks.
Docs look good. Removing the label.
Patch attached.
I am also in process of writing detailed documentation on the usage of this that we can add to the Hive wiki once this gets reviewed and merged to trunk.