Hive
  1. Hive
  2. HIVE-6147

Support avro data stored in HBase columns

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.12.0
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured.

      1. HIVE-6147.5.patch.txt
        211 kB
        Swarnim Kulkarni
      2. HIVE-6147.4.patch.txt
        208 kB
        Swarnim Kulkarni
      3. HIVE-6147.3.patch.txt
        208 kB
        Swarnim Kulkarni
      4. HIVE-6147.3.patch.txt
        208 kB
        Swarnim Kulkarni
      5. HIVE-6147.2.patch.txt
        208 kB
        Swarnim Kulkarni
      6. HIVE-6147.1.patch.txt
        205 kB
        Swarnim Kulkarni

        Activity

        Hide
        Xuefu Zhang added a comment -

        Swarnim Kulkarni I'm glad that you have the principle of code reuse in mind. I only browsed the patch, and spotted HiveSerdeHelper.getSchemaFromFS(), which is seemingly for the same purpose as AvroSerdeUtils.getSchemaFromFS() is. This might be coincidental. No big deal.

        Show
        Xuefu Zhang added a comment - Swarnim Kulkarni I'm glad that you have the principle of code reuse in mind. I only browsed the patch, and spotted HiveSerdeHelper.getSchemaFromFS(), which is seemingly for the same purpose as AvroSerdeUtils.getSchemaFromFS() is. This might be coincidental. No big deal.
        Hide
        Swarnim Kulkarni added a comment -

        Thanks Xuefu Zhang for reviewing. I agree it makes lot of sense for HIVE-6411 to go in first and then I can refactor this on the basis of that.

        Also on the point of reusing AvroSerDe code, I have tried to write AvroLazyObjectInspector simply as a wrapper on top of AvroSerDe still delegating most of the operations to the serde. Any specific instance P&C instance you want me to look deeper into?

        Show
        Swarnim Kulkarni added a comment - Thanks Xuefu Zhang for reviewing. I agree it makes lot of sense for HIVE-6411 to go in first and then I can refactor this on the basis of that. Also on the point of reusing AvroSerDe code, I have tried to write AvroLazyObjectInspector simply as a wrapper on top of AvroSerDe still delegating most of the operations to the serde. Any specific instance P&C instance you want me to look deeper into?
        Hide
        Xuefu Zhang added a comment -

        Thanks for working on this, Swarnim Kulkarni.

        1. This patch might have an ineligible overlap with HIVE-6411. Does it make sense to get that in first? Please work with Nevis regarding that.
        2. For the generated test code, please make sure that it will not make its way to the production jars.
        3. Avoid trailing spaces/tabs.
        4. Please reuse AvroSerde code rather than P&C.
        5. Avoid catching individual exception even if the handling of the exception is the same.

        Show
        Xuefu Zhang added a comment - Thanks for working on this, Swarnim Kulkarni . 1. This patch might have an ineligible overlap with HIVE-6411 . Does it make sense to get that in first? Please work with Nevis regarding that. 2. For the generated test code, please make sure that it will not make its way to the production jars. 3. Avoid trailing spaces/tabs. 4. Please reuse AvroSerde code rather than P&C. 5. Avoid catching individual exception even if the handling of the exception is the same.
        Hide
        Swarnim Kulkarni added a comment -

        Xuefu Zhang As the previously failing tests now pass, I have updated the RB with the latest patch for review.

        Show
        Swarnim Kulkarni added a comment - Xuefu Zhang As the previously failing tests now pass, I have updated the RB with the latest patch for review.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12633602/HIVE-6147.5.patch.txt

        ERROR: -1 due to 1 failed/errored test(s), 5381 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12633602

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633602/HIVE-6147.5.patch.txt ERROR: -1 due to 1 failed/errored test(s), 5381 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1691/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12633602
        Hide
        Swarnim Kulkarni added a comment -

        New patch rebased with master.

        Show
        Swarnim Kulkarni added a comment - New patch rebased with master.
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12633416/HIVE-6147.4.patch.txt

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1672/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java'
        ++ awk '{print $2}'
        ++ egrep -v '^X|^Performing status on external'
        ++ svn status --no-ignore
        + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/parquet_timestamp.q.out ql/src/test/results/clientnegative/parquet_char.q.out ql/src/test/results/clientnegative/parquet_date.q.out ql/src/test/results/clientnegative/parquet_decimal.q.out ql/src/test/results/clientnegative/parquet_varchar.q.out ql/src/test/queries/clientnegative/parquet_char.q ql/src/test/queries/clientnegative/parquet_timestamp.q ql/src/test/queries/clientnegative/parquet_decimal.q ql/src/test/queries/clientnegative/parquet_date.q ql/src/test/queries/clientnegative/parquet_varchar.q
        + svn update
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        External at revision 1575684.
        
        At revision 1575684.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12633416

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633416/HIVE-6147.4.patch.txt Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1672/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1672/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/parquet_timestamp.q.out ql/src/test/results/clientnegative/parquet_char.q.out ql/src/test/results/clientnegative/parquet_date.q.out ql/src/test/results/clientnegative/parquet_decimal.q.out ql/src/test/results/clientnegative/parquet_varchar.q.out ql/src/test/queries/clientnegative/parquet_char.q ql/src/test/queries/clientnegative/parquet_timestamp.q ql/src/test/queries/clientnegative/parquet_decimal.q ql/src/test/queries/clientnegative/parquet_date.q ql/src/test/queries/clientnegative/parquet_varchar.q + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1575684. At revision 1575684. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12633416
        Hide
        Swarnim Kulkarni added a comment -

        So I cleaned up my local m2 repository, rebased the master and re-applied the patch and saw the test failure. May be it was a bad dep sitting in my repo that was causing the problem.

        Attaching a new patch. If the tests pass with this, I will update the RB with this latest patch.

        Show
        Swarnim Kulkarni added a comment - So I cleaned up my local m2 repository, rebased the master and re-applied the patch and saw the test failure. May be it was a bad dep sitting in my repo that was causing the problem. Attaching a new patch. If the tests pass with this, I will update the RB with this latest patch.
        Hide
        Xuefu Zhang added a comment -

        Swarnim Kulkarni I'm not totally convinced that these tests are unrelated, as they consistently appeared in the test result. In addition, I manually ran TestHCatLoader, and got errors as the following:

        testProjectionsBasic(org.apache.hive.hcatalog.pig.TestHCatLoader)  Time elapsed: 0.184 sec  <<< ERROR!
        java.io.IOException: Failed to execute "create table junit_unparted_complex(name string, studentid int, contact struct<phno:string,email:string>, currently_registered_courses array<string>, current_grades map<string,string>, phnos array<struct<phno:string,type:string>>) stored as RCFILE tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver')". Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException
        	at org.apache.hive.hcatalog.pig.TestHCatLoader.executeStatementOnDriver(TestHCatLoader.java:125)
        	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:111)
        	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:101)
        	at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:115)
        	at org.apache.hive.hcatalog.pig.TestHCatLoader.setup(TestHCatLoader.java:154)
        

        Please further investigate.

        Show
        Xuefu Zhang added a comment - Swarnim Kulkarni I'm not totally convinced that these tests are unrelated, as they consistently appeared in the test result. In addition, I manually ran TestHCatLoader, and got errors as the following: testProjectionsBasic(org.apache.hive.hcatalog.pig.TestHCatLoader) Time elapsed: 0.184 sec <<< ERROR! java.io.IOException: Failed to execute "create table junit_unparted_complex(name string, studentid int , contact struct<phno:string,email:string>, currently_registered_courses array<string>, current_grades map<string,string>, phnos array<struct<phno:string,type:string>>) stored as RCFILE tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver')" . Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException at org.apache.hive.hcatalog.pig.TestHCatLoader.executeStatementOnDriver(TestHCatLoader.java:125) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:111) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:101) at org.apache.hive.hcatalog.pig.TestHCatLoader.createTable(TestHCatLoader.java:115) at org.apache.hive.hcatalog.pig.TestHCatLoader.setup(TestHCatLoader.java:154) Please further investigate.
        Hide
        Swarnim Kulkarni added a comment -

        To circle back on this one, I re-ran the tests locally again and they all seem to pass for me(See the screenshot above)

        Show
        Swarnim Kulkarni added a comment - To circle back on this one, I re-ran the tests locally again and they all seem to pass for me(See the screenshot above)
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12630595/HIVE-6147.3.patch.txt

        ERROR: -1 due to 25 failed/errored test(s), 5186 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
        org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
        org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
        org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 25 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12630595

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630595/HIVE-6147.3.patch.txt ERROR: -1 due to 25 failed/errored test(s), 5186 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1485/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12630595
        Hide
        Xuefu Zhang added a comment -

        Swarnim Kulkarni Right now the queue for test runs are long, and it may take a day or two before you see the result. I have just manually created a test job in the queue.

        Show
        Xuefu Zhang added a comment - Swarnim Kulkarni Right now the queue for test runs are long, and it may take a day or two before you see the result. I have just manually created a test job in the queue.
        Hide
        Swarnim Kulkarni added a comment -

        Xuefu Zhang Any way we can get the pre-commit tests running on this guy? Seems like with the latest attached patch, the test did not run.

        Show
        Swarnim Kulkarni added a comment - Xuefu Zhang Any way we can get the pre-commit tests running on this guy? Seems like with the latest attached patch, the test did not run.
        Hide
        Yong Zhang added a comment -

        I was looking for this feature, as Hbase itself is hard to handle 1 to M relationship, or support nest structure data, if we are only using Hbase as a DW storage. If we can store the AVRO bytes in one column family of Hbase table, and export that Avro schema out in the Hive, that can give us random update/insert the data as AVRO in Hbase, and use it in Hive or MR jobs generated from Hive.

        Show
        Yong Zhang added a comment - I was looking for this feature, as Hbase itself is hard to handle 1 to M relationship, or support nest structure data, if we are only using Hbase as a DW storage. If we can store the AVRO bytes in one column family of Hbase table, and export that Avro schema out in the Hive, that can give us random update/insert the data as AVRO in Hbase, and use it in Hive or MR jobs generated from Hive.
        Hide
        Swarnim Kulkarni added a comment -

        Reattaching patch to re-run the pre-commit tests on trunk to see if the results differ from my local run.

        Show
        Swarnim Kulkarni added a comment - Reattaching patch to re-run the pre-commit tests on trunk to see if the results differ from my local run.
        Hide
        Swarnim Kulkarni added a comment -

        I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed!

        mac-swarnim:hive swarnim$ git pull --rebase
        Current branch trunk is up to date.
        mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
        --2014-02-23 18:40:27--  https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt
        Resolving issues.apache.org... 140.211.11.121
        Connecting to issues.apache.org|140.211.11.121|:443... connected.
        HTTP request sent, awaiting response... 200 OK
        Length: 212743 (208K) [text/plain]
        Saving to: `HIVE-6147.3.patch.txt'
        
        100%[=====================================================================================================================>] 212,743     29.5K/s   in 7.0s    
        
        2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743]
        
        mac-swarnim:hive swarnim$ git status
        # On branch trunk
        # Untracked files:
        #   (use "git add <file>..." to include in what will be committed)
        #
        #	HIVE-6147.3.patch.txt
        nothing added to commit but untracked files present (use "git add" to track)
        mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt
        patching file hbase-handler/pom.xml
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java
        patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java
        patching file hbase-handler/src/if/avro/avro_test.avpr
        patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
        patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
        patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
        patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
        patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
        patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
        patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
        
        mac-swarnim:hive swarnim$ git status
        # On branch trunk
        # Changes not staged for commit:
        #   (use "git add <file>..." to update what will be committed)
        #   (use "git checkout -- <file>..." to discard changes in working directory)
        #
        #	modified:   hbase-handler/pom.xml
        #	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
        #	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        #	modified:   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
        #	modified:   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
        #	modified:   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
        #	modified:   serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
        #
        # Untracked files:
        #   (use "git add <file>..." to include in what will be committed)
        #
        #	HIVE-6147.3.patch.txt
        #	hbase-handler/src/gen/
        #	hbase-handler/src/if/
        #	hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java
        #	hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
        #	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
        #	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java
        #	serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
        
        mac-swarnim:hive swarnim$ cd hcatalog/
        mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1
        [INFO] Scanning for projects...
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter ---
        [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target
        [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = [])
        [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = [])
        [INFO] 
        [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter ---
        [debug] execute contextualize
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter ---
        [INFO] Executing tasks
        
        main:
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter ---
        [INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes
        [WARNING] Note: Some input files use or override a deprecated API.
        [WARNING] Note: Recompile with -Xlint:deprecation for details.
        [WARNING] Note: Some input files use unchecked or unsafe operations.
        [WARNING] Note: Recompile with -Xlint:unchecked for details.
        [INFO] 
        [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter ---
        [debug] execute contextualize
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter ---
        [INFO] Executing tasks
        
        main:
            [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp
            [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse
            [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
             [copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter ---
        [INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes
        [WARNING] Note: Some input files use or override a deprecated API.
        [WARNING] Note: Recompile with -Xlint:deprecation for details.
        [INFO] 
        [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter ---
        [INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hcatalog.pig.TestHCatLoader
        2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader
        Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
        2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema
        Running org.apache.hcatalog.pig.TestHCatLoaderStorer
        2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer
        Running org.apache.hcatalog.pig.TestHCatStorer
        2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore
        Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer
        Running org.apache.hcatalog.pig.TestHCatStorerMulti
        2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti
        Running org.apache.hcatalog.pig.TestHCatStorerWrapper
        2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper
        Running org.apache.hcatalog.pig.TestOrcHCatLoader
        2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader
        Running org.apache.hcatalog.pig.TestOrcHCatStorer
        2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer
        Running org.apache.hcatalog.pig.TestPigHCatUtil
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil
        Running org.apache.hive.hcatalog.pig.TestE2EScenarios
        2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios
        Running org.apache.hive.hcatalog.pig.TestHCatLoader
        2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore
        Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader
        Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
        2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
        Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
        2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
        Running org.apache.hive.hcatalog.pig.TestHCatStorer
        2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore
        Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer
        Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti
        2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti
        Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
        2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
        Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader
        2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore
        Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader
        Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
        2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema
        Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer
        2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer
        Running org.apache.hive.hcatalog.pig.TestPigHCatUtil
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil
        
        Results :
        
        Tests run: 101, Failures: 0, Errors: 0, Skipped: 0
        
        [INFO] 
        [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter ---
        [INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
        [INFO] 
        [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter ---
        [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar
        [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 9:11.629s
        [INFO] Finished at: Sun Feb 23 18:51:54 CST 2014
        [INFO] Final Memory: 24M/81M
        [INFO] ------------------------------------------------------------------------
        
        Show
        Swarnim Kulkarni added a comment - I am not sure what's going on here. I checked out the latest trunk, applied the patch and re-ran the failing tests locally and they all passed! mac-swarnim:hive swarnim$ git pull --rebase Current branch trunk is up to date. mac-swarnim:hive swarnim$ wget https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt --2014-02-23 18:40:27-- https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt Resolving issues.apache.org... 140.211.11.121 Connecting to issues.apache.org|140.211.11.121|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 212743 (208K) [text/plain] Saving to: `HIVE-6147.3.patch.txt' 100%[=====================================================================================================================>] 212,743 29.5K/s in 7.0s 2014-02-23 18:40:37 (29.5 KB/s) - `HIVE-6147.3.patch.txt' saved [212743/212743] mac-swarnim:hive swarnim$ git status # On branch trunk # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt nothing added to commit but untracked files present (use "git add" to track) mac-swarnim:hive swarnim$ patch -p0 < HIVE-6147.3.patch.txt patching file hbase-handler/pom.xml patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Address.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/ContactInfo.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Employee.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Gender.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/HomePhone.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/Magic.java patching file hbase-handler/src/gen/avro/gen-java/org/apache/hadoop/hive/hbase/avro/OfficePhone.java patching file hbase-handler/src/if/avro/avro_test.avpr patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java patching file hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java patching file hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java patching file serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java patching file serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java patching file serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java mac-swarnim:hive swarnim$ git status # On branch trunk # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: hbase-handler/pom.xml # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java # modified: hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java # modified: hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java # modified: serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # HIVE-6147.3.patch.txt # hbase-handler/src/gen/ # hbase-handler/src/if/ # hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java # hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java # serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java mac-swarnim:hive swarnim$ cd hcatalog/ mac-swarnim:hcatalog swarnim$ mvn clean install -pl hcatalog-pig-adapter -P hadoop-1 [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive HCatalog Pig Adapter 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-pig-adapter --- [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter (includes = [datanucleus.log, derby.log], excludes = []) [INFO] Deleting /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/build (includes = [], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 10 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hcatalog-pig-adapter --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-pig-adapter --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/warehouse [mkdir] Created dir: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [copy] Copying 5 files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hcatalog-pig-adapter --- [INFO] Compiling 26 source files to /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/test-classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-pig-adapter --- [INFO] Surefire report directory: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hcatalog.pig.TestHCatLoader 2014-02-23 16:42:49.909 java[65062:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.476 sec - in org.apache.hcatalog.pig.TestHCatLoader Running org.apache.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:43:45.590 java[65112:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.887 sec - in org.apache.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:44:14.765 java[65319:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hcatalog.pig.TestHCatStorer 2014-02-23 16:44:26.599 java[65328:1903] Unable to load realm info from SCDynamicStore Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.412 sec - in org.apache.hcatalog.pig.TestHCatStorer Running org.apache.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:44:47.482 java[65363:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.681 sec - in org.apache.hcatalog.pig.TestHCatStorerMulti Running org.apache.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:45:01.495 java[65377:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.07 sec - in org.apache.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:45:06.985 java[65384:1903] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.718 sec - in org.apache.hcatalog.pig.TestOrcHCatLoader Running org.apache.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:46:04.004 java[65438:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.293 sec - in org.apache.hcatalog.pig.TestOrcHCatStorer Running org.apache.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.hcatalog.pig.TestPigHCatUtil Running org.apache.hive.hcatalog.pig.TestE2EScenarios 2014-02-23 16:46:16.502 java[65454:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.143 sec - in org.apache.hive.hcatalog.pig.TestE2EScenarios Running org.apache.hive.hcatalog.pig.TestHCatLoader 2014-02-23 16:46:25.991 java[65461:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.261 sec - in org.apache.hive.hcatalog.pig.TestHCatLoader Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema 2014-02-23 16:47:30.547 java[65536:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.867 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer 2014-02-23 16:47:59.693 java[65744:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.167 sec - in org.apache.hive.hcatalog.pig.TestHCatLoaderStorer Running org.apache.hive.hcatalog.pig.TestHCatStorer 2014-02-23 16:48:11.201 java[65754:1903] Unable to load realm info from SCDynamicStore Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.175 sec - in org.apache.hive.hcatalog.pig.TestHCatStorer Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti 2014-02-23 16:49:50.434 java[65845:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.798 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerMulti Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper 2014-02-23 16:50:05.471 java[65860:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.097 sec - in org.apache.hive.hcatalog.pig.TestHCatStorerWrapper Running org.apache.hive.hcatalog.pig.TestOrcHCatLoader 2014-02-23 16:50:11.008 java[65866:1903] Unable to load realm info from SCDynamicStore Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.125 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoader Running org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema 2014-02-23 16:51:15.432 java[65940:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.779 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer 2014-02-23 16:51:42.577 java[66137:1903] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.179 sec - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer Running org.apache.hive.hcatalog.pig.TestPigHCatUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.513 sec - in org.apache.hive.hcatalog.pig.TestPigHCatUtil Results : Tests run: 101, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-pig-adapter --- [INFO] Building jar: /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hcatalog-pig-adapter --- [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/target/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.jar [INFO] Installing /Users/swarnim/git-repo/apache/hive/hcatalog/hcatalog-pig-adapter/pom.xml to /Users/swarnim/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-pig-adapter/0.13.0-SNAPSHOT/hive-hcatalog-pig-adapter-0.13.0-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9:11.629s [INFO] Finished at: Sun Feb 23 18:51:54 CST 2014 [INFO] Final Memory: 24M/81M [INFO] ------------------------------------------------------------------------
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt

        ERROR: -1 due to 24 failed/errored test(s), 5140 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
        org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
        org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 24 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12629556

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629556/HIVE-6147.3.patch.txt ERROR: -1 due to 24 failed/errored test(s), 5140 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1390/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed This message is automatically generated. ATTACHMENT ID: 12629556
        Hide
        Swarnim Kulkarni added a comment -

        Yeah. I did dig into the test report and the failures seem unrelated to the code that this patch touches.

        I have rebased the branch to master to see if it fixes anything. If these tests still fail, I'll start digging deeper.

        Show
        Swarnim Kulkarni added a comment - Yeah. I did dig into the test report and the failures seem unrelated to the code that this patch touches. I have rebased the branch to master to see if it fixes anything. If these tests still fail, I'll start digging deeper.
        Hide
        Xuefu Zhang added a comment -

        Swarnim Kulkarni Thanks for submitting the patch. Have you investigated the test failures shown above?

        Show
        Xuefu Zhang added a comment - Swarnim Kulkarni Thanks for submitting the patch. Have you investigated the test failures shown above?
        Hide
        Swarnim Kulkarni added a comment -

        Brock Noland,Xuefu Zhang If one of you get a chance to do a quick review of this, I would really appreciate that.

        Thanks,

        Show
        Swarnim Kulkarni added a comment - Brock Noland , Xuefu Zhang If one of you get a chance to do a quick review of this, I would really appreciate that. Thanks,
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12626225/HIVE-6147.2.patch.txt

        ERROR: -1 due to 21 failed/errored test(s), 4987 tests executed
        Failed tests:

        org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
        org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 21 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12626225

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626225/HIVE-6147.2.patch.txt ERROR: -1 due to 21 failed/errored test(s), 4987 tests executed Failed tests: org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1131/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed This message is automatically generated. ATTACHMENT ID: 12626225
        Hide
        Swarnim Kulkarni added a comment -
        Show
        Swarnim Kulkarni added a comment - Review Request: https://reviews.apache.org/r/17566/
        Hide
        Swarnim Kulkarni added a comment -

        Xuefu Zhang You are correct. This adds support to the existing HBaseSerDe via ObjectInspectors to query avro data stored in HBase. Also you make a very good point about having a unified interface to allow for the querying of any structured format stored in any storage like HBase or HDFS. Specifically we can look at creating specific implementations of the HBase DataType[1] to have such layer on top of HBase. The idea here though this specific problem and get this support in for now and then move towards a more generalized approach.

        [1] https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/types/DataType.html

        Show
        Swarnim Kulkarni added a comment - Xuefu Zhang You are correct. This adds support to the existing HBaseSerDe via ObjectInspectors to query avro data stored in HBase. Also you make a very good point about having a unified interface to allow for the querying of any structured format stored in any storage like HBase or HDFS. Specifically we can look at creating specific implementations of the HBase DataType [1] to have such layer on top of HBase. The idea here though this specific problem and get this support in for now and then move towards a more generalized approach. [1] https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/types/DataType.html
        Hide
        Swarnim Kulkarni added a comment -

        Attaching new patch with very minor formatting updates and adding missing license headers.

        Show
        Swarnim Kulkarni added a comment - Attaching new patch with very minor formatting updates and adding missing license headers.
        Hide
        Xuefu Zhang added a comment -

        This looks good, but from the patch, it seems that the solution is only for HBase. I wonder if we have given thoughts on the idea of generalizing the problem and providing a general solution. I can see the benefits of separating the storage (such as hbase) and data format (avro, thrift, protocol buf, parquet, etc). Then we solve M + N problems rather than M * N problems. What if the avro data is coming from other storage, such as accumulo, or parquet data from HBase.

        Show
        Xuefu Zhang added a comment - This looks good, but from the patch, it seems that the solution is only for HBase. I wonder if we have given thoughts on the idea of generalizing the problem and providing a general solution. I can see the benefits of separating the storage (such as hbase) and data format (avro, thrift, protocol buf, parquet, etc). Then we solve M + N problems rather than M * N problems. What if the avro data is coming from other storage, such as accumulo, or parquet data from HBase.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12626065/HIVE-6147.1.patch.txt

        ERROR: -1 due to 22 failed/errored test(s), 4987 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
        org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct
        org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt
        org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
        org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 22 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12626065

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626065/HIVE-6147.1.patch.txt ERROR: -1 due to 22 failed/errored test(s), 4987 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby org.apache.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hcatalog.pig.TestHCatStorer.testBagNStruct org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1119/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12626065
        Hide
        Swarnim Kulkarni added a comment -

        Patch attached.

        I am also in process of writing detailed documentation on the usage of this that we can add to the Hive wiki once this gets reviewed and merged to trunk.

        Show
        Swarnim Kulkarni added a comment - Patch attached. I am also in process of writing detailed documentation on the usage of this that we can add to the Hive wiki once this gets reviewed and merged to trunk.

          People

          • Assignee:
            Swarnim Kulkarni
            Reporter:
            Swarnim Kulkarni
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development