Hive
  1. Hive
  2. HIVE-6147

Support avro data stored in HBase columns

        Details

        • Type: Bug Bug
        • Status: Patch Available
        • Priority: Major Major
        • Resolution: Unresolved
        • Affects Version/s: 0.12.0
        • Fix Version/s: None
        • Component/s: HBase Handler
        • Labels:
          None

          Description

          Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured.

          1. Text File
            HIVE-6147.5.patch.txt
            211 kB
            Swarnim Kulkarni
          2. Text File
            HIVE-6147.4.patch.txt
            208 kB
            Swarnim Kulkarni
          3. Text File
            HIVE-6147.3.patch.txt
            208 kB
            Swarnim Kulkarni
          4. Text File
            HIVE-6147.3.patch.txt
            208 kB
            Swarnim Kulkarni
          5. Text File
            HIVE-6147.2.patch.txt
            208 kB
            Swarnim Kulkarni
          6. Text File
            HIVE-6147.1.patch.txt
            205 kB
            Swarnim Kulkarni

            Activity

            Descending order - Click to sort in ascending order
            Hide
            Permalink
            Xuefu Zhang added a comment -

            Swarnim Kulkarni I'm glad that you have the principle of code reuse in mind. I only browsed the patch, and spotted HiveSerdeHelper.getSchemaFromFS(), which is seemingly for the same purpose as AvroSerdeUtils.getSchemaFromFS() is. This might be coincidental. No big deal.

            Show
            Xuefu Zhang added a comment - Swarnim Kulkarni I'm glad that you have the principle of code reuse in mind. I only browsed the patch, and spotted HiveSerdeHelper.getSchemaFromFS(), which is seemingly for the same purpose as AvroSerdeUtils.getSchemaFromFS() is. This might be coincidental. No big deal.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Thanks Xuefu Zhang for reviewing. I agree it makes lot of sense for HIVE-6411 to go in first and then I can refactor this on the basis of that.

            Also on the point of reusing AvroSerDe code, I have tried to write AvroLazyObjectInspector simply as a wrapper on top of AvroSerDe still delegating most of the operations to the serde. Any specific instance P&C instance you want me to look deeper into?

            Show
            Swarnim Kulkarni added a comment - Thanks Xuefu Zhang for reviewing. I agree it makes lot of sense for HIVE-6411 to go in first and then I can refactor this on the basis of that. Also on the point of reusing AvroSerDe code, I have tried to write AvroLazyObjectInspector simply as a wrapper on top of AvroSerDe still delegating most of the operations to the serde. Any specific instance P&C instance you want me to look deeper into?
            Hide
            Permalink
            Xuefu Zhang added a comment -

            Thanks for working on this, Swarnim Kulkarni.

            1. This patch might have an ineligible overlap with HIVE-6411. Does it make sense to get that in first? Please work with Nevis regarding that.
            2. For the generated test code, please make sure that it will not make its way to the production jars.
            3. Avoid trailing spaces/tabs.
            4. Please reuse AvroSerde code rather than P&C.
            5. Avoid catching individual exception even if the handling of the exception is the same.

            Show
            Xuefu Zhang added a comment - Thanks for working on this, Swarnim Kulkarni . 1. This patch might have an ineligible overlap with HIVE-6411 . Does it make sense to get that in first? Please work with Nevis regarding that. 2. For the generated test code, please make sure that it will not make its way to the production jars. 3. Avoid trailing spaces/tabs. 4. Please reuse AvroSerde code rather than P&C. 5. Avoid catching individual exception even if the handling of the exception is the same.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Xuefu Zhang As the previously failing tests now pass, I have updated the RB with the latest patch for review.

            Show
            Swarnim Kulkarni added a comment - Xuefu Zhang As the previously failing tests now pass, I have updated the RB with the latest patch for review.
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12633602/HIVE-6147.5.patch.txt

            ERROR: -1 due to 1 failed/errored test(s), 5381 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

            This message is automatically generated.

            ATTACHMENT ID: 12633602

            Show
            Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633602/HIVE-6147.5.patch.txt ERROR: -1 due to 1 failed/errored test(s), 5381 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12633602
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            New patch rebased with master.

            Show
            Swarnim Kulkarni added a comment - New patch rebased with master.
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 no tests executed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12633416/HIVE-6147.4.patch.txt

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1672/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/parquet_timestamp.q.out ql/src/test/results/clientnegative/parquet_char.q.out ql/src/test/results/clientnegative/parquet_date.q.out ql/src/test/results/clientnegative/parquet_decimal.q.out ql/src/test/results/clientnegative/parquet_varchar.q.out ql/src/test/queries/clientnegative/parquet_char.q ql/src/test/queries/clientnegative/parquet_timestamp.q ql/src/test/queries/clientnegative/parquet_decimal.q ql/src/test/queries/clientnegative/parquet_date.q ql/src/test/queries/clientnegative/parquet_varchar.q
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1575684.

At revision 1575684.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'

            This message is automatically generated.

            ATTACHMENT ID: 12633416

            Show
            Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633416/HIVE-6147.4.patch.txt Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1672/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/parquet_timestamp.q.out ql/src/test/results/clientnegative/parquet_char.q.out ql/src/test/results/clientnegative/parquet_date.q.out ql/src/test/results/clientnegative/parquet_decimal.q.out ql/src/test/results/clientnegative/parquet_varchar.q.out ql/src/test/queries/clientnegative/parquet_char.q ql/src/test/queries/clientnegative/parquet_timestamp.q ql/src/test/queries/clientnegative/parquet_decimal.q ql/src/test/queries/clientnegative/parquet_date.q ql/src/test/queries/clientnegative/parquet_varchar.q + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1575684. At revision 1575684. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12633416
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            So I cleaned up my local m2 repository, rebased the master and re-applied the patch and saw the test failure. May be it was a bad dep sitting in my repo that was causing the problem.

            Attaching a new patch. If the tests pass with this, I will update the RB with this latest patch.

            Show
            Swarnim Kulkarni added a comment - So I cleaned up my local m2 repository, rebased the master and re-applied the patch and saw the test failure. May be it was a bad dep sitting in my repo that was causing the problem. Attaching a new patch. If the tests pass with this, I will update the RB with this latest patch.
            Hide
            Permalink
            Xuefu Zhang added a comment -

            Swarnim Kulkarni I'm not totally convinced that these tests are unrelated, as they consistently appeared in the test result. In addition, I manually ran TestHCatLoader, and got errors as the following:

            testProjectionsBasic(org.apache.hive.hcatalog.pig.TestHCatLoader)  Time elapsed: 0.184 sec  <<< ERROR!
java.io.IOException: Failed to execute "create table junit_unparted_complex(name string, studentid int, contact struct<phno:string,email:string>, currently_registered_courses array<string>, current_grades map<string,string>, phnos array<struct<phno:string,type:string>>) stored as RCFILE tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver')". Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException
	at org.apache.hive.hcatalog.pig.TestHCatLoader.executeStatementOnDriver(TestHCatLoader.java:125)
	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:111)
	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:101)
	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:115)
	at org.apache.hive.hcatalog.pig.TestHCatLoader.setup(TestHCatLoader.java:154)

            Please further investigate.

            Show
            Xuefu Zhang added a comment - Swarnim Kulkarni I'm not totally convinced that these tests are unrelated, as they consistently appeared in the test result. In addition, I manually ran TestHCatLoader, and got errors as the following: testProjectionsBasic(org.apache.hive.hcatalog.pig.TestHCatLoader) Time elapsed: 0.184 sec <<< ERROR! java.io.IOException: Failed to execute "create table junit_unparted_complex(name string, studentid int , contact struct<phno:string,email:string>, currently_registered_courses array<string>, current_grades map<string,string>, phnos array<struct<phno:string,type:string>>) stored as RCFILE tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver')" . Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException at org.apache.hive.hcatalog.pig.TestHCatLoader.executeStatementOnDriver(TestHCatLoader.java:125) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:111) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:101) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:115) at org.apache.hive.hcatalog.pig.TestHCatLoader.setup(TestHCatLoader.java:154) Please further investigate.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            To circle back on this one, I re-ran the tests locally again and they all seem to pass for me(See the screenshot above)

            Show
            Swarnim Kulkarni added a comment - To circle back on this one, I re-ran the tests locally again and they all seem to pass for me(See the screenshot above)
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12630595/HIVE-6147.3.patch.txt

            ERROR: -1 due to 25 failed/errored test(s), 5186 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed

            This message is automatically generated.

            ATTACHMENT ID: 12630595

            Show
            Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630595/HIVE-6147.3.patch.txt ERROR: -1 due to 25 failed/errored test(s), 5186 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12630595
            Hide
            Permalink
            Xuefu Zhang added a comment -

            Swarnim Kulkarni Right now the queue for test runs are long, and it may take a day or two before you see the result. I have just manually created a test job in the queue.

            Show
            Xuefu Zhang added a comment - Swarnim Kulkarni Right now the queue for test runs are long, and it may take a day or two before you see the result. I have just manually created a test job in the queue.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Xuefu Zhang Any way we can get the pre-commit tests running on this guy? Seems like with the latest attached patch, the test did not run.

            Show
            Swarnim Kulkarni added a comment - Xuefu Zhang Any way we can get the pre-commit tests running on this guy? Seems like with the latest attached patch, the test did not run.
            Hide
            Permalink
            Yong Zhang added a comment -

            I was looking for this feature, as Hbase itself is hard to handle 1 to M relationship, or support nest structure data, if we are only using Hbase as a DW storage. If we can store the AVRO bytes in one column family of Hbase table, and export that Avro schema out in the Hive, that can give us random update/insert the data as AVRO in Hbase, and use it in Hive or MR jobs generated from Hive.

            Show
            Yong Zhang added a comment - I was looking for this feature, as Hbase itself is hard to handle 1 to M relationship, or support nest structure data, if we are only using Hbase as a DW storage. If we can store the AVRO bytes in one column family of Hbase table, and export that Avro schema out in the Hive, that can give us random update/insert the data as AVRO in Hbase, and use it in Hive or MR jobs generated from Hive.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Reattaching patch to re-run the pre-commit tests on trunk to see if the results differ from my local run.

            Show
            Swarnim Kulkarni added a comment - Reattaching patch to re-run the pre-commit tests on trunk to see if the results differ from my local run.
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed!

            mac-swarnim:hive swarnim$ git pull --rebase
Current branch trunk is up to date.
mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
--2014-02-23 18:40:27--  https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
Resolving issues.apache.org... 140.211.11.121
Connecting to issues.apache.org|140.211.11.121|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 212743 (208K) [text/plain]
Saving to: `HIVE-6147.3.patch.txt'

100%[=====================================================================================================================>] 212,743     29.5K/s   in 7.0s    

2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743]

mac-swarnim:hive swarnim$ git status
# On branch trunk
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	HIVE-6147.3.patch.txt
nothing added to commit but untracked files present (use "git add" to track)
mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt
patching file hbase-handler/pom.xml
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java
patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java
patching file hbase-handler/src/if/avro/avro_test.avpr
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java

mac-swarnim:hive swarnim$ git status
# On branch trunk
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   hbase-handler/pom.xml
#	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
#	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
#	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
#	modified:   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
#	modified:   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
#	modified:   serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	HIVE-6147.3.patch.txt
#	hbase-handler/src/gen/
#	hbase-handler/src/if/
#	hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
#	hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
#	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
#	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
#	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java

mac-swarnim:hive swarnim$ cd hcatalog/
mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter ---
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = [])
[INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter ---
[INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp
    [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse
    [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
     [copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter ---
[INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter ---
[INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hcatalog.pig.TestHCatLoader
2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader
Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
Running org.apache.hcatalog.pig.TestHCatLoaderStorer
2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer
Running org.apache.hcatalog.pig.TestHCatStorer
2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer
Running org.apache.hcatalog.pig.TestHCatStorerMulti
2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti
Running org.apache.hcatalog.pig.TestHCatStorerWrapper
2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper
Running org.apache.hcatalog.pig.TestOrcHCatLoader
2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader
Running org.apache.hcatalog.pig.TestOrcHCatStorer
2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer
Running org.apache.hcatalog.pig.TestPigHCatUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil
Running org.apache.hive.hcatalog.pig.TestE2EScenarios
2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios
Running org.apache.hive.hcatalog.pig.TestHCatLoader
2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader
Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorer
2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti
2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti
Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader
2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader
Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer
2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer
Running org.apache.hive.hcatalog.pig.TestPigHCatUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil

Results :

Tests run: 101, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter ---
[INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter ---
[INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
[INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9:11.629s
[INFO] Finished at: Sun Feb 23 18:51:54 CST 2014
[INFO] Final Memory: 24M/81M
[INFO] ------------------------------------------------------------------------
            Show
            Swarnim Kulkarni added a comment - I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed! mac-swarnim:hive swarnim$ git pull --rebase Current branch trunk is up to date. mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt --2014-02-23 18:40:27-- https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt Resolving issues.apache.org... 140.211.11.121 Connecting to issues.apache.org|140.211.11.121|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 212743 (208K) [text/plain] Saving to: `HIVE-6147.3.patch.txt' 100%[=====================================================================================================================>] 212,743 29.5K/s in 7.0s 2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743] mac-swarnim:hive swarnim$ git status # On branch trunk # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt nothing added to commit but untracked files present (use "git add" to track) mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt patching file hbase-handler/pom.xml patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java patching file hbase-handler/src/if/avro/avro_test.avpr patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java mac-swarnim:hive swarnim$ git status # On branch trunk # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: hbase-handler/pom.xml # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt # hbase-handler/src/gen/ # hbase-handler/src/if/ # hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java # hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java mac-swarnim:hive swarnim$ cd hcatalog/ mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1 [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter --- [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = []) [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter --- [INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hcatalog.pig.TestHCatLoader 2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hcatalog.pig.TestHCatStorer 2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer Running org.apache.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti Running org.apache.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader Running org.apache.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer Running org.apache.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil Running org.apache.hive.hcatalog.pig.TestE2EScenarios 2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios Running org.apache.hive.hcatalog.pig.TestHCatLoader 2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hive.hcatalog.pig.TestHCatStorer 2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema 2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer Running org.apache.hive.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil Results : Tests run: 101, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter --- [INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter --- [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9:11.629s [INFO] Finished at: Sun Feb 23 18:51:54 CST 2014 [INFO] Final Memory: 24M/81M [INFO] ------------------------------------------------------------------------
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt

            ERROR: -1 due to 24 failed/errored test(s), 5140 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed

            This message is automatically generated.

            ATTACHMENT ID: 12629556

            Show
            Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt ERROR: -1 due to 24 failed/errored test(s), 5140 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed This message is automatically generated. ATTACHMENT ID: 12629556
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Yeah. I did dig into the test report and the failures seem unrelated to the code that this patch touches.

            I have rebased the branch to master to see if it fixes anything. If these tests still fail, I'll start digging deeper.

            Show
            Swarnim Kulkarni added a comment - Yeah. I did dig into the test report and the failures seem unrelated to the code that this patch touches. I have rebased the branch to master to see if it fixes anything. If these tests still fail, I'll start digging deeper.
            Hide
            Permalink
            Xuefu Zhang added a comment -

            Swarnim Kulkarni Thanks for submitting the patch. Have you investigated the test failures shown above?

            Show
            Xuefu Zhang added a comment - Swarnim Kulkarni Thanks for submitting the patch. Have you investigated the test failures shown above?
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Brock Noland,Xuefu Zhang If one of you get a chance to do a quick review of this, I would really appreciate that.

            Thanks,

            Show
            Swarnim Kulkarni added a comment - Brock Noland , Xuefu Zhang If one of you get a chance to do a quick review of this, I would really appreciate that. Thanks,
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12626225/HIVE-6147.2.patch.txt

            ERROR: -1 due to 21 failed/errored test(s), 4987 tests executed
            Failed tests:

            org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed

            This message is automatically generated.

            ATTACHMENT ID: 12626225

            Show
            Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626225/HIVE-6147.2.patch.txt ERROR: -1 due to 21 failed/errored test(s), 4987 tests executed Failed tests: org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed This message is automatically generated. ATTACHMENT ID: 12626225
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Review Request: https://reviews.apache.org/r/17566/

            Show
            Swarnim Kulkarni added a comment - Review Request: https://reviews.apache.org/r/17566/
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Xuefu Zhang You are correct. This adds support to the existing HBaseSerDe via ObjectInspectors to query avro data stored in HBase. Also you make a very good point about having a unified interface to allow for the querying of any structured format stored in any storage like HBase or HDFS. Specifically we can look at creating specific implementations of the HBase DataType[1] to have such layer on top of HBase. The idea here though this specific problem and get this support in for now and then move towards a more generalized approach.

            [1] https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/types/DataType.html

            Show
            Swarnim Kulkarni added a comment - Xuefu Zhang You are correct. This adds support to the existing HBaseSerDe via ObjectInspectors to query avro data stored in HBase. Also you make a very good point about having a unified interface to allow for the querying of any structured format stored in any storage like HBase or HDFS. Specifically we can look at creating specific implementations of the HBase DataType [1] to have such layer on top of HBase. The idea here though this specific problem and get this support in for now and then move towards a more generalized approach. [1] https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/types/DataType.html
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Attaching new patch with very minor formatting updates and adding missing license headers.

            Show
            Swarnim Kulkarni added a comment - Attaching new patch with very minor formatting updates and adding missing license headers.
            Hide
            Permalink
            Xuefu Zhang added a comment -

            This looks good, but from the patch, it seems that the solution is only for HBase. I wonder if we have given thoughts on the idea of generalizing the problem and providing a general solution. I can see the benefits of separating the storage (such as hbase) and data format (avro, thrift, protocol buf, parquet, etc). Then we solve M + N problems rather than M * N problems. What if the avro data is coming from other storage, such as accumulo, or parquet data from HBase.

            Show
            Xuefu Zhang added a comment - This looks good, but from the patch, it seems that the solution is only for HBase. I wonder if we have given thoughts on the idea of generalizing the problem and providing a general solution. I can see the benefits of separating the storage (such as hbase) and data format (avro, thrift, protocol buf, parquet, etc). Then we solve M + N problems rather than M * N problems. What if the avro data is coming from other storage, such as accumulo, or parquet data from HBase.
            Hide
            Permalink
            Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12626065/HIVE-6147.1.patch.txt

            ERROR: -1 due to 22 failed/errored test(s), 4987 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed

            This message is automatically generated.

            ATTACHMENT ID: 12626065

            Show
            Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626065/HIVE-6147.1.patch.txt ERROR: -1 due to 22 failed/errored test(s), 4987 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12626065
            Hide
            Permalink
            Swarnim Kulkarni added a comment -

            Patch attached.

            I am also in process of writing detailed documentation on the usage of this that we can add to the Hive wiki once this gets reviewed and merged to trunk.

            Show
            Swarnim Kulkarni added a comment - Patch attached. I am also in process of writing detailed documentation on the usage of this that we can add to the Hive wiki once this gets reviewed and merged to trunk.

              People

              • Assignee:
                Swarnim Kulkarni
                Reporter:
                Swarnim Kulkarni
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Development