Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      Add support for filters to be provided via the composite key class

      1. HIVE-6290.1.patch.txt
        17 kB
        Swarnim Kulkarni
      2. HIVE-6290.2.patch.txt
        20 kB
        Swarnim Kulkarni
      3. HIVE-6290.3.patch.txt
        18 kB
        Swarnim Kulkarni

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Xuefu Zhang added a comment -

          User doc should go with HIVE-6411 also.

          Show
          Xuefu Zhang added a comment - User doc should go with HIVE-6411 also.
          Hide
          Lefty Leverenz added a comment -

          Does this need any user doc?

          Show
          Lefty Leverenz added a comment - Does this need any user doc?
          Hide
          Xuefu Zhang added a comment -

          Resolved via HIVE-6411.

          Show
          Xuefu Zhang added a comment - Resolved via HIVE-6411 .
          Hide
          Navis added a comment -

          I've regarded that as a following issue but I've merged your patch into HIVE-6411.

          Show
          Navis added a comment - I've regarded that as a following issue but I've merged your patch into HIVE-6411 .
          Hide
          Swarnim Kulkarni added a comment -

          Thanks Navis. So your proposal here is that we first let HIVE-6411 in and then apply changes made in this patch on top of it or kill the patch here all together in lieu for HIVE-6411?

          Show
          Swarnim Kulkarni added a comment - Thanks Navis . So your proposal here is that we first let HIVE-6411 in and then apply changes made in this patch on top of it or kill the patch here all together in lieu for HIVE-6411 ?
          Hide
          Navis added a comment -

          Maybe I can make HBaseCompositeKey$HBaseCompositeKeyFactory to implement HBaseWritableKeyFactory, which is

          public interface HBaseWritableKeyFactory extends HBaseKeyFactory, HiveStoragePredicateHandler {
            boolean serialize(Object object, ObjectInspector inspector, ByteStream.Output output) throws IOException;
          }
          
          Show
          Navis added a comment - Maybe I can make HBaseCompositeKey$HBaseCompositeKeyFactory to implement HBaseWritableKeyFactory, which is public interface HBaseWritableKeyFactory extends HBaseKeyFactory, HiveStoragePredicateHandler { boolean serialize( Object object, ObjectInspector inspector, ByteStream.Output output) throws IOException; }
          Hide
          Swarnim Kulkarni added a comment -

          Navis Out of curiosity, do you have a rough design proposal for this support over HIVE-6411? Just was curious as to how things could change. Thanks.

          Show
          Swarnim Kulkarni added a comment - Navis Out of curiosity, do you have a rough design proposal for this support over HIVE-6411 ? Just was curious as to how things could change. Thanks.
          Hide
          Xuefu Zhang added a comment -

          Navis Thanks for sharing your thoughts. I haven't looked at HIVE-6411, so not sure of the relationship between the two. I'll take a look and figured out what's the best. Thanks again.

          Show
          Xuefu Zhang added a comment - Navis Thanks for sharing your thoughts. I haven't looked at HIVE-6411 , so not sure of the relationship between the two. I'll take a look and figured out what's the best. Thanks again.
          Hide
          Navis added a comment -

          Swarnim Kulkarni Xuefu Zhang I'm really sorry to cut in. But could you consider HIVE-6411 before this? IMHO, this kind of functionality could be provided in more generic way, including full support of predicate pushdown, etc. Just consider once.

          Show
          Navis added a comment - Swarnim Kulkarni Xuefu Zhang I'm really sorry to cut in. But could you consider HIVE-6411 before this? IMHO, this kind of functionality could be provided in more generic way, including full support of predicate pushdown, etc. Just consider once.
          Hide
          Xuefu Zhang added a comment -

          Patch looks good. Minor comment on review board.

          Show
          Xuefu Zhang added a comment - Patch looks good. Minor comment on review board.
          Hide
          Swarnim Kulkarni added a comment -

          Done.

          Show
          Swarnim Kulkarni added a comment - Done.
          Hide
          Xuefu Zhang added a comment -

          Swarnim Kulkarni The review board (https://reviews.apache.org/r/17483/) doesn't seem having your latest code change. Do you mind updating that? You can reuse the same review request, but uploading diff with your latest patch.

          Show
          Xuefu Zhang added a comment - Swarnim Kulkarni The review board ( https://reviews.apache.org/r/17483/ ) doesn't seem having your latest code change. Do you mind updating that? You can reuse the same review request, but uploading diff with your latest patch.
          Hide
          Swarnim Kulkarni added a comment -

          Brock NolandXuefu Zhang Unless there is something like you guys want me to look at with this patch, this should be ready to be merged.

          Show
          Swarnim Kulkarni added a comment - Brock Noland Xuefu Zhang Unless there is something like you guys want me to look at with this patch, this should be ready to be merged.
          Hide
          Swarnim Kulkarni added a comment -

          Thanks NAvis. Brock Noland I have addressed all the review comments. Anything else you want me to look at for this JIRA?

          Show
          Swarnim Kulkarni added a comment - Thanks NAvis. Brock Noland I have addressed all the review comments. Anything else you want me to look at for this JIRA?
          Hide
          Navis added a comment -

          Swarnim Kulkarni Interesting. I think I've missed how to make filter predicates (and how to write to hbase) in HIVE-6411. We can proceed with this patch as is state and then I should extend HIVE-6411 functionality onto it.

          Show
          Navis added a comment - Swarnim Kulkarni Interesting. I think I've missed how to make filter predicates (and how to write to hbase) in HIVE-6411 . We can proceed with this patch as is state and then I should extend HIVE-6411 functionality onto it.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12629551/HIVE-6290.3.patch.txt

          SUCCESS: +1 5133 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1389/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1389/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12629551

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629551/HIVE-6290.3.patch.txt SUCCESS: +1 5133 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1389/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1389/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12629551
          Hide
          Swarnim Kulkarni added a comment -

          Navis Would you also mind taking a quick look as it is related to some of the work that you have been doing with HIVE-6411.

          Show
          Swarnim Kulkarni added a comment - Navis Would you also mind taking a quick look as it is related to some of the work that you have been doing with HIVE-6411 .
          Hide
          Swarnim Kulkarni added a comment -

          Addressed comments. New patch attached.

          Show
          Swarnim Kulkarni added a comment - Addressed comments. New patch attached.
          Hide
          Brock Noland added a comment - - edited

          Nice work! I see you created two RB items for this patch. Perhaps you know this already and created a new item for some other reason, but you can update an existing RB item with a the latest version of a patch.

          Thank you very much for your contributions!!

          Show
          Brock Noland added a comment - - edited Nice work! I see you created two RB items for this patch. Perhaps you know this already and created a new item for some other reason, but you can update an existing RB item with a the latest version of a patch. Thank you very much for your contributions!!
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12625799/HIVE-6290.2.patch.txt

          SUCCESS: +1 4972 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1096/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1096/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12625799

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625799/HIVE-6290.2.patch.txt SUCCESS: +1 4972 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1096/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1096/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12625799
          Hide
          Swarnim Kulkarni added a comment -

          Fixed and new patch added.

          Updated review request: https://reviews.apache.org/r/17483/

          Show
          Swarnim Kulkarni added a comment - Fixed and new patch added. Updated review request: https://reviews.apache.org/r/17483/
          Hide
          Swarnim Kulkarni added a comment -

          Absolutely. Sorry for that. I was under the impression that the eclipse formatter would do it for me. But HIVE-6317 was the result of me realizing that it wasn't. I'll get that fixed.

          Show
          Swarnim Kulkarni added a comment - Absolutely. Sorry for that. I was under the impression that the eclipse formatter would do it for me. But HIVE-6317 was the result of me realizing that it wasn't. I'll get that fixed.
          Hide
          Brock Noland added a comment -

          Hi,

          I noticed there are some tabs in the file. Could you change the tabs to spaces?

          Brock

          Show
          Brock Noland added a comment - Hi, I noticed there are some tabs in the file. Could you change the tabs to spaces? Brock
          Hide
          Brock Noland added a comment -

          Hi Swarnim,

          Yes those tests are failing on trunk at present. There is work going on to fix them.

          Thank you!!
          Brock

          Show
          Brock Noland added a comment - Hi Swarnim, Yes those tests are failing on trunk at present. There is work going on to fix them. Thank you!! Brock
          Hide
          Swarnim Kulkarni added a comment -

          The test failures here seem unrelated. The test pass locally on my machine.

          mac-swarnim:itests sk018283$ git branch
          * issue/HIVE-6290
            trunk
          
          mac-swarnim:itests sk018283$ mvn test -Dtest=TestMinimrCliDriver -Dqfile=testCliDriver_load_hdfs_file_with_space_in_the_name.q -Dtest.output.overwrite=true -Phadoop-1
          [INFO] Scanning for projects...
          [INFO] ------------------------------------------------------------------------
          [INFO] Hive Integration - QFile Tests
          ......
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hive.cli.TestMinimrCliDriver
          2014-01-26 20:36:16.574 java[68243:1903] Unable to load realm info from SCDynamicStore
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 226.088 sec - in org.apache.hadoop.hive.cli.TestMinimrCliDriver
          
          Results :
          
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          
          Show
          Swarnim Kulkarni added a comment - The test failures here seem unrelated. The test pass locally on my machine. mac-swarnim:itests sk018283$ git branch * issue/HIVE-6290 trunk mac-swarnim:itests sk018283$ mvn test -Dtest=TestMinimrCliDriver -Dqfile=testCliDriver_load_hdfs_file_with_space_in_the_name.q -Dtest.output.overwrite=true -Phadoop-1 [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Hive Integration - QFile Tests ...... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hive.cli.TestMinimrCliDriver 2014-01-26 20:36:16.574 java[68243:1903] Unable to load realm info from SCDynamicStore Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 226.088 sec - in org.apache.hadoop.hive.cli.TestMinimrCliDriver Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12625241/HIVE-6290.1.patch.txt

          ERROR: -1 due to 3 failed/errored test(s), 4958 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1033/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1033/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12625241

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625241/HIVE-6290.1.patch.txt ERROR: -1 due to 3 failed/errored test(s), 4958 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1033/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1033/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12625241
          Hide
          Swarnim Kulkarni added a comment -
          Show
          Swarnim Kulkarni added a comment - Review request: https://reviews.apache.org/r/17370/

            People

            • Assignee:
              Swarnim Kulkarni
              Reporter:
              Swarnim Kulkarni
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development