Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11609

Capability to add a filter to hbase scan via composite key doesn't work

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      It seems like the capability to add filter to an hbase scan which was added as part of HIVE-6411 doesn't work. This is primarily because in the HiveHBaseInputFormat, the filter is added in the getsplits instead of getrecordreader. This works fine for start and stop keys but not for filter because a filter is respected only when an actual scan is performed. This is also related to the initial refactoring that was done as part of HIVE-3420.

      1. HIVE-11609.08.patch
        23 kB
        Barna Zsombor Klara
      2. HIVE-11609.09.patch
        24 kB
        Barna Zsombor Klara
      3. HIVE-11609.1.patch.txt
        21 kB
        Swarnim Kulkarni
      4. HIVE-11609.2.patch.txt
        25 kB
        Swarnim Kulkarni
      5. HIVE-11609.3.patch.txt
        26 kB
        Swarnim Kulkarni
      6. HIVE-11609.4.patch.txt
        25 kB
        Swarnim Kulkarni
      7. HIVE-11609.5.patch
        25 kB
        Yongzhi Chen
      8. HIVE-11609.6.patch.txt
        24 kB
        Swarnim Kulkarni
      9. HIVE-11609.7.patch.txt
        22 kB
        Swarnim Kulkarni

        Issue Links

          Activity

          Hide
          ychena Yongzhi Chen added a comment -

          Ashutosh Chauhan, I am not very familiar with that part of code, could you help to review it? Thanks

          Show
          ychena Yongzhi Chen added a comment - Ashutosh Chauhan , I am not very familiar with that part of code, could you help to review it? Thanks
          Hide
          ychena Yongzhi Chen added a comment -

          Barna Zsombor Klara, I am not very comfortable with the change related to handle existing filters. The filter got from property
          TableScanDesc.FILTER_OBJECT_CONF_STR should include all the needed filter information to the table, to include the existing filters in the scan object may redundant or may cause issue (for example the pass-in scan object contains out-of-date filters. ) ?

          Show
          ychena Yongzhi Chen added a comment - Barna Zsombor Klara , I am not very comfortable with the change related to handle existing filters. The filter got from property TableScanDesc.FILTER_OBJECT_CONF_STR should include all the needed filter information to the table, to include the existing filters in the scan object may redundant or may cause issue (for example the pass-in scan object contains out-of-date filters. ) ?
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          Swarnim Kulkarni, we need a +1 from a committer.
          Yongzhi Chen are you fine with the current patch or should we ask for someone more familiar with hbase to take a look?
          FYI in my opinion the test failure was due to an incorrect test case that was working before the patch only because the filtering was not being done on HBase.

          Show
          zsombor.klara Barna Zsombor Klara added a comment - Swarnim Kulkarni , we need a +1 from a committer. Yongzhi Chen are you fine with the current patch or should we ask for someone more familiar with hbase to take a look? FYI in my opinion the test failure was due to an incorrect test case that was working before the patch only because the filtering was not being done on HBase.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Barna Zsombor Klara Thanks a lot for taking a look into this.

          +1 on the patch from me. Anything else we need to do to get this merged in?

          Show
          swarnim Swarnim Kulkarni added a comment - Barna Zsombor Klara Thanks a lot for taking a look into this. +1 on the patch from me. Anything else we need to do to get this merged in?
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12866869/HIVE-11609.09.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 1 failed/errored test(s), 10654 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5112/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5112/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5112/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12866869 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12866869/HIVE-11609.09.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 1 failed/errored test(s), 10654 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5112/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5112/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5112/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12866869 - PreCommit-HIVE-Build
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          Regenerated the golden file for hbase_custom_key3 and reuploaded the patch.

          Show
          zsombor.klara Barna Zsombor Klara added a comment - Regenerated the golden file for hbase_custom_key3 and reuploaded the patch.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12866754/HIVE-11609.08.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 2 failed/errored test(s), 10652 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3] (batchId=90)
          org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.testPartitionPublish (batchId=180)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5096/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5096/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5096/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12866754 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12866754/HIVE-11609.08.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 2 failed/errored test(s), 10652 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3] (batchId=90) org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.testPartitionPublish (batchId=180) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5096/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5096/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5096/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12866754 - PreCommit-HIVE-Build
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          Reuploading patch as the precommit job didn't pick it up.

          Show
          zsombor.klara Barna Zsombor Klara added a comment - Reuploading patch as the precommit job didn't pick it up.
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          Minor modifications, the core of the patch should be the same. The only difference is that I'm trying to handle existing filters in the HBaseScanRange and I removed the equality filter from the SampleHBaseKeyFactory3.

          Show
          zsombor.klara Barna Zsombor Klara added a comment - Minor modifications, the core of the patch should be the same. The only difference is that I'm trying to handle existing filters in the HBaseScanRange and I removed the equality filter from the SampleHBaseKeyFactory3.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Hey Barna Zsombor Klara. Please feel free to take a stab if you have time. Unfortunately I am very occupied for next few weeks but can definitely help answer any questions.

          Show
          swarnim Swarnim Kulkarni added a comment - Hey Barna Zsombor Klara . Please feel free to take a stab if you have time. Unfortunately I am very occupied for next few weeks but can definitely help answer any questions.
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          Swarnim Kulkarni, would you like to update the patch to have all tests passing or if you are busy, then I can finish it if you don't mind.

          Show
          zsombor.klara Barna Zsombor Klara added a comment - Swarnim Kulkarni , would you like to update the patch to have all tests passing or if you are busy, then I can finish it if you don't mind.
          Hide
          zsombor.klara Barna Zsombor Klara added a comment -

          I had a discussion with Balazs Meszaros about the failing unit test hbase_custom_key3. It seems that our implementation for the SampleHBaseKeyFactory3 was incorrect to begin with, because the equality operator cannot be used to filter parts of a custom key in HBase. The test was passing before this change because the filter wasn't taken into account anyway, and now it is failing because we are trying to filter for keys equal to '238' and '1238' at the same time. Based on this I would say that Swarnim Kulkarni was correct to remove that line from SampleHBaseKeyFactory3.
          I have however two other concerns (may be irrelevant, I'm not familiar with the hbase-handler in Hive):

          • in SampleHBaseKeyFactory3 I think the filterlist operator should be 'MUST_PASS_ALL'
            if (!filters.isEmpty()) {
                  range.addFilter(new FilterList(FilterList.Operator.MUST_PASS_ALL, filters));
                }
            
          • in HBaseScanRange#setup we are setting the filter on the scan object without checking if there are preexistent filters already present, should't we check for existing filters and add them together with the newly deserialised ones or are we sure (for reasons unclear to me) that the scan will not contain any?
          Show
          zsombor.klara Barna Zsombor Klara added a comment - I had a discussion with Balazs Meszaros about the failing unit test hbase_custom_key3. It seems that our implementation for the SampleHBaseKeyFactory3 was incorrect to begin with, because the equality operator cannot be used to filter parts of a custom key in HBase. The test was passing before this change because the filter wasn't taken into account anyway, and now it is failing because we are trying to filter for keys equal to '238' and '1238' at the same time. Based on this I would say that Swarnim Kulkarni was correct to remove that line from SampleHBaseKeyFactory3 . I have however two other concerns (may be irrelevant, I'm not familiar with the hbase-handler in Hive): in SampleHBaseKeyFactory3 I think the filterlist operator should be 'MUST_PASS_ALL' if (!filters.isEmpty()) { range.addFilter( new FilterList(FilterList.Operator.MUST_PASS_ALL, filters)); } in HBaseScanRange#setup we are setting the filter on the scan object without checking if there are preexistent filters already present, should't we check for existing filters and add them together with the newly deserialised ones or are we sure (for reasons unclear to me) that the scan will not contain any?
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12862366/HIVE-11609.7.patch.txt

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 5 failed/errored test(s), 10579 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=235)
          org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] (batchId=235)
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3] (batchId=90)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143)
          org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4602/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12862366 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12862366/HIVE-11609.7.patch.txt ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 5 failed/errored test(s), 10579 tests executed Failed tests: org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=235) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] (batchId=235) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3] (batchId=90) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4602/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12862366 - PreCommit-HIVE-Build
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Rebased and updated patch.

          Show
          swarnim Swarnim Kulkarni added a comment - Rebased and updated patch.
          Hide
          ychena Yongzhi Chen added a comment -

          From test history, org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 is still failing.

          Show
          ychena Yongzhi Chen added a comment - From test history, org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 is still failing.
          Hide
          aihuaxu Aihua Xu added a comment - - edited

          Swarnim Kulkarni The patch has been there for a while. Can you rebase your fix to the latest code and reattach it? Let me know if you will be able to work on it.

          Show
          aihuaxu Aihua Xu added a comment - - edited Swarnim Kulkarni The patch has been there for a while. Can you rebase your fix to the latest code and reattach it? Let me know if you will be able to work on it.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Can I get one of the hive committers to take a quick look at this one?

          Show
          swarnim Swarnim Kulkarni added a comment - Can I get one of the hive committers to take a quick look at this one?
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12797338/HIVE-11609.6.patch.txt

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 59 failed/errored test(s), 9896 tests executed
          Failed tests:

          TestMiniTezCliDriver-tez_smb_empty.q-mapjoin_decimal.q-transform_ppr2.q-and-12-more - did not produce a TEST-*.xml file
          TestMiniTezCliDriver-vector_acid3.q-vector_decimal_trailing.q-lvj_mapjoin.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
          org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
          org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
          org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
          org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
          org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
          org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
          org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
          org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
          org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
          org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleTable
          org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTable
          org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
          org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
          org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
          org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
          org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
          org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
          org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
          org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
          org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
          org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
          org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
          org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
          org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
          org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
          org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
          org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
          org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
          org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
          org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
          org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
          org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat
          org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
          org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
          org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
          org.apache.hive.spark.client.TestSparkClient.testSyncRpc
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7508/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7508/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7508/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 59 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12797338 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12797338/HIVE-11609.6.patch.txt ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 59 failed/errored test(s), 9896 tests executed Failed tests: TestMiniTezCliDriver-tez_smb_empty.q-mapjoin_decimal.q-transform_ppr2.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_acid3.q-vector_decimal_trailing.q-lvj_mapjoin.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleTable org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTable org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS org.apache.hive.spark.client.TestSparkClient.testSyncRpc Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7508/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7508/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7508/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 59 tests failed This message is automatically generated. ATTACHMENT ID: 12797338 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12797105/HIVE-11609.6.patch.txt

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 2 failed/errored test(s), 9976 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7486/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7486/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7486/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12797105 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12797105/HIVE-11609.6.patch.txt ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 2 failed/errored test(s), 9976 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7486/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7486/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7486/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12797105 - PreCommit-HIVE-TRUNK-Build
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Yongzhi Chen Sorry for the late reply on this. I added that code back to the test file as you suggested and have all the test passing locally as well[1].

          Attaching the latest patch to be run on Jenkins.

          [1] http://pastebin.com/gX52nivF

          Show
          swarnim Swarnim Kulkarni added a comment - Yongzhi Chen Sorry for the late reply on this. I added that code back to the test file as you suggested and have all the test passing locally as well [1] . Attaching the latest patch to be run on Jenkins. [1] http://pastebin.com/gX52nivF
          Hide
          ychena Yongzhi Chen added a comment -

          Swarnim Kulkarni, I did a test, after I added UDFOPEqual back to the code, the hbase_custom_key3.q will produce the same query plan as without the fix. But there will be no value returns for query select * from hbase_ck_5 where key.col1 = '238' AND key.col2 = '1238'
          The row

          {"col1":"238","col2":"1238","col3":"2238"}

          val_238 is not returned.
          Hope that help you remember how to explain the change.

          Show
          ychena Yongzhi Chen added a comment - Swarnim Kulkarni , I did a test, after I added UDFOPEqual back to the code, the hbase_custom_key3.q will produce the same query plan as without the fix. But there will be no value returns for query select * from hbase_ck_5 where key.col1 = '238' AND key.col2 = '1238' The row {"col1":"238","col2":"1238","col3":"2238"} val_238 is not returned. Hope that help you remember how to explain the change.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Yongzhi Chen Thanks for taking a look. Unfortunately I am not 100% sure at this point why but looking at the sequence of patches it looks like that was in to address a test failure that was caused by the first patch.

          Show
          swarnim Swarnim Kulkarni added a comment - Yongzhi Chen Thanks for taking a look. Unfortunately I am not 100% sure at this point why but looking at the sequence of patches it looks like that was in to address a test failure that was caused by the first patch.
          Hide
          ychena Yongzhi Chen added a comment -

          +1 pending on Swarnim Kulkarni answering my previous question related to UDFOPEqual (although it is in the test code, I would like to know why).
          For I made some change to the patch, Szehon Ho or Aihua Xu, could you also review the code? Thanks

          Show
          ychena Yongzhi Chen added a comment - +1 pending on Swarnim Kulkarni answering my previous question related to UDFOPEqual (although it is in the test code, I would like to know why). For I made some change to the patch, Szehon Ho or Aihua Xu , could you also review the code? Thanks
          Hide
          ychena Yongzhi Chen added a comment -

          The 4 failures are not related. testNegativeCliDriver_authorization_uri_import and testSSLVersion failed hundreds of times.
          The MiniTez no test xml is just a random failure.
          Ran TestTxnCommands2 locally, all pass:

          Running org.apache.hadoop.hive.ql.TestTxnCommands2
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 108.325 sec - in org.apache.hadoop.hive.ql.TestTxnCommands2
          
          Show
          ychena Yongzhi Chen added a comment - The 4 failures are not related. testNegativeCliDriver_authorization_uri_import and testSSLVersion failed hundreds of times. The MiniTez no test xml is just a random failure. Ran TestTxnCommands2 locally, all pass: Running org.apache.hadoop.hive.ql.TestTxnCommands2 Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 108.325 sec - in org.apache.hadoop.hive.ql.TestTxnCommands2
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12784444/HIVE-11609.5.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 4 failed/errored test(s), 10017 tests executed
          Failed tests:

          TestMiniTezCliDriver-auto_sortmerge_join_13.q-alter_merge_2_orc.q-vectorized_parquet_types.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
          org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
          org.apache.hive.jdbc.TestSSL.testSSLVersion
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6755/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6755/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6755/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12784444 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12784444/HIVE-11609.5.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 4 failed/errored test(s), 10017 tests executed Failed tests: TestMiniTezCliDriver-auto_sortmerge_join_13.q-alter_merge_2_orc.q-vectorized_parquet_types.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6755/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6755/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6755/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12784444 - PreCommit-HIVE-TRUNK-Build
          Hide
          ychena Yongzhi Chen added a comment -

          For your reference, the stack caused by the AbstractHBaseKeyPredicateDecomposer.java is

          2016-01-26T07:53:00,458 ERROR [7000edc7-3285-4e38-bd88-8115ff9f0d0a main[]]: parse.CalcitePlanner (CalcitePlanner.java:genOPTree(307)) - CBO failed, skipping CBO. 
          org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Unexpected rexnode : org.apache.calcite.rex.RexInputRef
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:151) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:136) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:202) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:130) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:202) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:130) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterRelNode(CalcitePlanner.java:1710) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterRelNode(CalcitePlanner.java:1862) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterLogicalPlan(CalcitePlanner.java:1895) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:3040) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:874) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:832) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:112) ~[calcite-core-1.5.0.jar:1.5.0]
                  at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:971) ~[calcite-core-1.5.0.jar:1.5.0]
                  at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:148) ~[calcite-core-1.5.0.jar:1.5.0]
                  at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105) ~[calcite-core-1.5.0.jar:1.5.0]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:655) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:270) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10100) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:472) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1168) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1256) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1094) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1082) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) [hive-cli-2.1.0-SNAPSHOT.jar:?]
                  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) [hive-cli-2.1.0-SNAPSHOT.jar:?]
                  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) [hive-cli-2.1.0-SNAPSHOT.jar:?]
                  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) [hive-cli-2.1.0-SNAPSHOT.jar:?]
                  at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1129) [hive-it-util-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1103) [hive-it-util-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
                  at org.apache.hadoop.hive.cli.TestHBaseCliDriver.runTest(TestHBaseCliDriver.java:92) [test-classes/:?]
                  at org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3(TestHBaseCliDriver.java:74) [test-classes/:?]
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_65]
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_65]
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_65]
                  at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_65]
                  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) [junit-4.11.jar:?]
                  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.11.jar:?]
                  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) [junit-4.11.jar:?]
                  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.11.jar:?]
                  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.11.jar:?]
          
          
          Show
          ychena Yongzhi Chen added a comment - For your reference, the stack caused by the AbstractHBaseKeyPredicateDecomposer.java is 2016-01-26T07:53:00,458 ERROR [7000edc7-3285-4e38-bd88-8115ff9f0d0a main[]]: parse.CalcitePlanner (CalcitePlanner.java:genOPTree(307)) - CBO failed, skipping CBO. org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Unexpected rexnode : org.apache.calcite.rex.RexInputRef at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:151) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:136) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:202) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:130) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:202) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter.convert(RexNodeConverter.java:130) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterRelNode(CalcitePlanner.java:1710) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterRelNode(CalcitePlanner.java:1862) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genFilterLogicalPlan(CalcitePlanner.java:1895) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:3040) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:874) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:832) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:112) ~[calcite-core-1.5.0.jar:1.5.0] at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:971) ~[calcite-core-1.5.0.jar:1.5.0] at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:148) ~[calcite-core-1.5.0.jar:1.5.0] at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105) ~[calcite-core-1.5.0.jar:1.5.0] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:655) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:270) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10100) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:472) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1168) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1256) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1094) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1082) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) [hive-cli-2.1.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) [hive-cli-2.1.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) [hive-cli-2.1.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) [hive-cli-2.1.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1129) [hive-it-util-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1103) [hive-it-util-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] at org.apache.hadoop.hive.cli.TestHBaseCliDriver.runTest(TestHBaseCliDriver.java:92) [test-classes/:?] at org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3(TestHBaseCliDriver.java:74) [test-classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_65] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_65] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_65] at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_65] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) [junit-4.11.jar:?] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.11.jar:?] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) [junit-4.11.jar:?] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.11.jar:?] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.11.jar:?]
          Hide
          ychena Yongzhi Chen added a comment -

          Swarnim Kulkarni, your extra change for AbstractHBaseKeyPredicateDecomposer.java since patch 3 causes the failures. I removed that change, the two tests all pass:
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
          I attach patch 5 which does not have the change for AbstractHBaseKeyPredicateDecomposer.java

          Show
          ychena Yongzhi Chen added a comment - Swarnim Kulkarni , your extra change for AbstractHBaseKeyPredicateDecomposer.java since patch 3 causes the failures. I removed that change, the two tests all pass: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler I attach patch 5 which does not have the change for AbstractHBaseKeyPredicateDecomposer.java
          Hide
          ychena Yongzhi Chen added a comment -

          Swarnim Kulkarni, have you checked the failures? And could you tell me when you make change to HBaseScanRange getScanRange(List<IndexSearchCondition> searchConditions), why do you remove the case for if (comparisonOp.endsWith("UDFOPEqual")) ? Thanks

          Show
          ychena Yongzhi Chen added a comment - Swarnim Kulkarni , have you checked the failures? And could you tell me when you make change to HBaseScanRange getScanRange(List<IndexSearchCondition> searchConditions), why do you remove the case for if (comparisonOp.endsWith("UDFOPEqual")) ? Thanks
          Hide
          ychena Yongzhi Chen added a comment -

          The 3 hbase test failures are related to the change, Swarnim Kulkarni, could you have a look?
          And when you attach a new patch, it may take a while before the pre-commit build run because there are many patch changes waiting in the queue and each build takes several hours.

          Show
          ychena Yongzhi Chen added a comment - The 3 hbase test failures are related to the change, Swarnim Kulkarni , could you have a look? And when you attach a new patch, it may take a while before the pre-commit build run because there are many patch changes waiting in the queue and each build takes several hours.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12781061/HIVE-11609.4.patch.txt

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 16 failed/errored test(s), 9982 tests executed
          Failed tests:

          TestHWISessionManager - did not produce a TEST-*.xml file
          TestMiniTezCliDriver-tez_joins_explain.q-vector_decimal_aggregate.q-vector_groupby_mapjoin.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
          org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
          org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
          org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
          org.apache.hive.jdbc.TestSSL.testSSLVersion
          org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
          org.apache.hive.spark.client.TestSparkClient.testCounters
          org.apache.hive.spark.client.TestSparkClient.testErrorJob
          org.apache.hive.spark.client.TestSparkClient.testJobSubmission
          org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
          org.apache.hive.spark.client.TestSparkClient.testRemoteClient
          org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
          org.apache.hive.spark.client.TestSparkClient.testSyncRpc
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6562/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 16 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12781061 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12781061/HIVE-11609.4.patch.txt ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 16 failed/errored test(s), 9982 tests executed Failed tests: TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-tez_joins_explain.q-vector_decimal_aggregate.q-vector_groupby_mapjoin.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles org.apache.hive.spark.client.TestSparkClient.testCounters org.apache.hive.spark.client.TestSparkClient.testErrorJob org.apache.hive.spark.client.TestSparkClient.testJobSubmission org.apache.hive.spark.client.TestSparkClient.testMetricsCollection org.apache.hive.spark.client.TestSparkClient.testRemoteClient org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob org.apache.hive.spark.client.TestSparkClient.testSyncRpc Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6562/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed This message is automatically generated. ATTACHMENT ID: 12781061 - PreCommit-HIVE-TRUNK-Build
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Yongzhi Chen Anyway to get the tests back running? Seems like reattaching the patch did not kick off the precommit build.

          Show
          swarnim Swarnim Kulkarni added a comment - Yongzhi Chen Anyway to get the tests back running? Seems like reattaching the patch did not kick off the precommit build.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Rebased and attached new patch to rerun tests. I however have been observing some flakiness locally after rebasing. Hopefully this should help debug if they fail on jenkins as well.

          Show
          swarnim Swarnim Kulkarni added a comment - Rebased and attached new patch to rerun tests. I however have been observing some flakiness locally after rebasing. Hopefully this should help debug if they fail on jenkins as well.
          Hide
          ychena Yongzhi Chen added a comment -

          Swarnim Kulkarni, how about the rebase? Could you upload the new patch? Thanks

          Show
          ychena Yongzhi Chen added a comment - Swarnim Kulkarni , how about the rebase? Could you upload the new patch? Thanks
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Thanks for looking into this Yongzhi Chen. It could also be because this patch is currently outdated with the master and I am not getting any of these failures locally. Let me try and rebase this with the master and see if that helps.

          Show
          swarnim Swarnim Kulkarni added a comment - Thanks for looking into this Yongzhi Chen . It could also be because this patch is currently outdated with the master and I am not getting any of these failures locally. Let me try and rebase this with the master and see if that helps.
          Hide
          ychena Yongzhi Chen added a comment -

          Swarnim Kulkarni, could you check the precommit tests failures, for example TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          may be related. Thanks

          Show
          ychena Yongzhi Chen added a comment - Swarnim Kulkarni , could you check the precommit tests failures, for example TestHBaseCliDriver.testCliDriver_hbase_custom_key3 may be related. Thanks
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12765499/HIVE-11609.3.patch.txt

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 4 failed/errored test(s), 9640 tests executed
          Failed tests:

          TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
          org.apache.hive.jdbc.TestSSL.testSSLVersion
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5581/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5581/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5581/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12765499 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12765499/HIVE-11609.3.patch.txt ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 4 failed/errored test(s), 9640 tests executed Failed tests: TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5581/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5581/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5581/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12765499 - PreCommit-HIVE-TRUNK-Build
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Ashutosh Chauhan Mind giving this another quick look and let me know if my comment here makes sense?

          Show
          swarnim Swarnim Kulkarni added a comment - Ashutosh Chauhan Mind giving this another quick look and let me know if my comment here makes sense?
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Reattaching patch rebasing with master and very minor updates.

          Show
          swarnim Swarnim Kulkarni added a comment - Reattaching patch rebasing with master and very minor updates.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Ashutosh Chauhan So I looked into this a little bit and looks like the fix with HIVE-10940 isn't really going to work, mostly because it seems to be pretty tailored to Tez. The SerializeFilter is only called for TezCompiler[1]. Any reason why this is not called for MapreduceCompiler too or SparkCompiler? As a quick hack, I tried calling it within the MapReduceCompiler in the "optimizeTaskPlan" but that doesn't seem to work very well too. Might need to dig a little bit into what's going on there. If it's ok with you, I would potentially like to log a separate bug though and tackle it there just to keep it separate from what we are trying to do here. If that works, we can re-add the "transient" and only go the SerializeFilter route.

          [1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java#L486-L490

          Show
          swarnim Swarnim Kulkarni added a comment - Ashutosh Chauhan So I looked into this a little bit and looks like the fix with HIVE-10940 isn't really going to work, mostly because it seems to be pretty tailored to Tez. The SerializeFilter is only called for TezCompiler [1] . Any reason why this is not called for MapreduceCompiler too or SparkCompiler? As a quick hack, I tried calling it within the MapReduceCompiler in the "optimizeTaskPlan" but that doesn't seem to work very well too. Might need to dig a little bit into what's going on there. If it's ok with you, I would potentially like to log a separate bug though and tackle it there just to keep it separate from what we are trying to do here. If that works, we can re-add the "transient" and only go the SerializeFilter route. [1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java#L486-L490
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Ahh. I see what is going on here. So I built this patch off a distribution[1] and tested it out. The distribution apparently did not have HIVE-10940 in it and hence no SerializeFilter class that you refer to. Which is why I had to remove the transient to be able to serialize the filter object. I looked through HIVE-10940 and seems like that should eliminate the need to do this. I am going to try this class and see if it fits the bill. Nice one Ashutosh Chauhan!

          [1] https://github.com/cloudera/hive/tree/cdh5.4.5-release/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical

          Show
          swarnim Swarnim Kulkarni added a comment - Ahh. I see what is going on here. So I built this patch off a distribution [1] and tested it out. The distribution apparently did not have HIVE-10940 in it and hence no SerializeFilter class that you refer to. Which is why I had to remove the transient to be able to serialize the filter object. I looked through HIVE-10940 and seems like that should eliminate the need to do this. I am going to try this class and see if it fits the bill. Nice one Ashutosh Chauhan ! [1] https://github.com/cloudera/hive/tree/cdh5.4.5-release/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          It is serialized in o.a.h.h.ql.optimizer.physical.SerializerFilter.java

          Show
          ashutoshc Ashutosh Chauhan added a comment - It is serialized in o.a.h.h.ql.optimizer.physical.SerializerFilter.java
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Hm. If that is the case, I think it should have worked without this change. Not 100% sure why it fails my testing then if I take the transient out by giving a null for the value for that field. Can you point me to the class with the serializedFilterObject and I will dig deeper.

          Show
          swarnim Swarnim Kulkarni added a comment - Hm. If that is the case, I think it should have worked without this change. Not 100% sure why it fails my testing then if I take the transient out by giving a null for the value for that field. Can you point me to the class with the serializedFilterObject and I will dig deeper.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Didn't follow you completely, but serializedFilterObject is not transient and in getRR() you will have a job conf from where you can retrieve it. So, that should work.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Didn't follow you completely, but serializedFilterObject is not transient and in getRR() you will have a job conf from where you can retrieve it. So, that should work.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          It would current read the scan object from the jobConf and my understanding was that with the way Kryo works with Hive, lot of this ser and deser happens via this jobConf and is controlled here[1]. So in order for that object to get into jobConf, the field had to be "de-transitionized".

          [1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L787-L798

          Show
          swarnim Swarnim Kulkarni added a comment - It would current read the scan object from the jobConf and my understanding was that with the way Kryo works with Hive, lot of this ser and deser happens via this jobConf and is controlled here [1] . So in order for that object to get into jobConf, the field had to be "de-transitionized". [1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L787-L798
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          We already have serializedFilterObject in TS. Can't that be used ?

          Show
          ashutoshc Ashutosh Chauhan added a comment - We already have serializedFilterObject in TS. Can't that be used ?
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Thanks a lot Ashutosh Chauhan for taking the time. I apologize for not getting back earlier. Been pretty swamped with other things.

          you have removed transient modifier on filterObject. Any reason for that?

          Yes. That is because the scan filters are now correctly handled inside the getRecordReader method unlike previously how they were handled in getSplits. Unlike the start and stop keys on a scan, a scan filter cannot be used to prune out regions before hand but is actually used when the scan actually runs. So when the individual tasks run, they read the filterObject that was previously serialized by hive as part of its execution plan and try to retrieve the scan filter out of it. With the transient in place, hive was serializing everything but the filter object which was need. So I had to modify that code to remove the "transient" so those objects can be serialized as well and be available to the scan.

          Show
          swarnim Swarnim Kulkarni added a comment - Thanks a lot Ashutosh Chauhan for taking the time. I apologize for not getting back earlier. Been pretty swamped with other things. you have removed transient modifier on filterObject. Any reason for that? Yes. That is because the scan filters are now correctly handled inside the getRecordReader method unlike previously how they were handled in getSplits. Unlike the start and stop keys on a scan, a scan filter cannot be used to prune out regions before hand but is actually used when the scan actually runs. So when the individual tasks run, they read the filterObject that was previously serialized by hive as part of its execution plan and try to retrieve the scan filter out of it. With the transient in place, hive was serializing everything but the filter object which was need. So I had to modify that code to remove the "transient" so those objects can be serialized as well and be available to the scan.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          I looked more into this. Hive actually removes the predicate once storage handler starts accepting it. Here, HBaseSH can now process those filters, so Hive removed it. Very much in line with what you observed. So, changes look good.
          One last question, in TableScanDesc, you have removed transient modifier on filterObject. Any reason for that?

          Show
          ashutoshc Ashutosh Chauhan added a comment - I looked more into this. Hive actually removes the predicate once storage handler starts accepting it. Here, HBaseSH can now process those filters, so Hive removed it. Very much in line with what you observed. So, changes look good. One last question, in TableScanDesc, you have removed transient modifier on filterObject. Any reason for that?
          Hide
          swarnim Swarnim Kulkarni added a comment -

          In one of .q tests, following line is removed :
          filterExpr: ((key.col1 = '238') and (key.col2 = '1238')) (type: boolean)
          which indicates filter was not pushed to TableScanOp.

          That not really true. With this issue I also found that it seems like the pushdown predicates were getting handled twice, once by the storagehandler and other by hive when they should only get handled by one of them(probably should log another bug for that). So the tests were passing entirely because hive was handling the predicates. The predicates were not even getting converted to the hbase filter. After this fix, the test composite key factory implementation passed to the query will start handling the predicates. That said, I am not entirely sure at this point how that line actually got removed. I'll take a look.

          Show
          swarnim Swarnim Kulkarni added a comment - In one of .q tests, following line is removed : filterExpr: ((key.col1 = '238') and (key.col2 = '1238')) (type: boolean) which indicates filter was not pushed to TableScanOp. That not really true. With this issue I also found that it seems like the pushdown predicates were getting handled twice, once by the storagehandler and other by hive when they should only get handled by one of them(probably should log another bug for that). So the tests were passing entirely because hive was handling the predicates. The predicates were not even getting converted to the hbase filter. After this fix, the test composite key factory implementation passed to the query will start handling the predicates. That said, I am not entirely sure at this point how that line actually got removed. I'll take a look.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          In one of .q tests, following line is removed :

          filterExpr: ((key.col1 = '238') and (key.col2 = '1238')) (type: boolean)

          which indicates filter was not pushed to TableScanOp. Is that expected ?

          Show
          ashutoshc Ashutosh Chauhan added a comment - In one of .q tests, following line is removed : filterExpr: ((key.col1 = '238') and (key.col2 = '1238')) (type: boolean) which indicates filter was not pushed to TableScanOp. Is that expected ?
          Hide
          sershe Sergey Shelukhin added a comment -

          +0.9. Ideally someone familiar with this code should take a look. Otherwise I will add 0.1 tomorrow
          Navis Ashutosh Chauhan

          Show
          sershe Sergey Shelukhin added a comment - +0.9. Ideally someone familiar with this code should take a look. Otherwise I will add 0.1 tomorrow Navis Ashutosh Chauhan
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Nick Dimiduk When you have a chance, would you be able to take a quick look?

          Show
          swarnim Swarnim Kulkarni added a comment - Nick Dimiduk When you have a chance, would you be able to take a quick look?
          Hide
          sershe Sergey Shelukhin added a comment -

          some feedback on RB. Looks good in general, but I'm not really familiar with that code. Nick Dimiduk do you want to take a look?

          Show
          sershe Sergey Shelukhin added a comment - some feedback on RB. Looks good in general, but I'm not really familiar with that code. Nick Dimiduk do you want to take a look?
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12753230/HIVE-11609.2.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 9381 tests executed
          Failed tests:

          org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
          org.apache.hive.hcatalog.streaming.TestStreaming.testRemainingTransactions
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5120/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5120/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5120/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12753230 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12753230/HIVE-11609.2.patch.txt ERROR: -1 due to 2 failed/errored test(s), 9381 tests executed Failed tests: org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.hcatalog.streaming.TestStreaming.testRemainingTransactions Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5120/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5120/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5120/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12753230 - PreCommit-HIVE-TRUNK-Build
          Hide
          swarnim Swarnim Kulkarni added a comment -

          RB request updated with latest patch.

          Show
          swarnim Swarnim Kulkarni added a comment - RB request updated with latest patch.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Some more info on my test environment:

          1. My keys were salted which means no hot regions and even key distribution across regions.
          2. No pre-splits when loading data. Though I feel the performance would have been even better if the regions were pre-split.
          3. My CompositeKeyFactory implementation would take the successive predicates(key1="something" and key2="something2") and convert them to apt scan filter. So results on those types of queries completely depends on how you are handing the predicate to filter conversion logic inside your custom implementation.

          Show
          swarnim Swarnim Kulkarni added a comment - Some more info on my test environment: 1. My keys were salted which means no hot regions and even key distribution across regions. 2. No pre-splits when loading data. Though I feel the performance would have been even better if the regions were pre-split. 3. My CompositeKeyFactory implementation would take the successive predicates(key1="something" and key2="something2") and convert them to apt scan filter. So results on those types of queries completely depends on how you are handing the predicate to filter conversion logic inside your custom implementation.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Updating patch to address the failing test.

          Show
          swarnim Swarnim Kulkarni added a comment - Updating patch to address the failing test.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12753168/HIVE-11609.1.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 9380 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
          org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5117/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5117/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5117/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12753168 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12753168/HIVE-11609.1.patch.txt ERROR: -1 due to 2 failed/errored test(s), 9380 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5117/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5117/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5117/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12753168 - PreCommit-HIVE-TRUNK-Build
          Show
          swarnim Swarnim Kulkarni added a comment - RB: https://reviews.apache.org/r/37930/
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Sergey ShelukhinGopal V Navis Mind taking a look at this? One of the things that I had to do to get this working is to serialize the filterObject in TableScanDesc. This seems to be related to HIVE-10940.

          Show
          swarnim Swarnim Kulkarni added a comment - Sergey Shelukhin Gopal V Navis Mind taking a look at this? One of the things that I had to do to get this working is to serialize the filterObject in TableScanDesc. This seems to be related to HIVE-10940 .
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Here are results from my testing with and without this patch applied. The table "my_table" for this testing contains about 8 M rows.

          Restrict query by single key:

          Example query: select * from my_table where key.firstpart="something";

          Memory(in MB) With patch Without patch
          1500 Out of memory Out of memory
          3000 2.5 minutes Out of memory
          6000 2.4 minutes 23 minutes

          Restrict query by multiple key: (Note that the key parts must be successive for this to work)

          Example query: select * from my_table where key.firstpart="something" and key.secondpart="something2";

          Memory(in MB) With filter Without filter
          1500 23 sec Out of memory
          3000 19 sec Out of memory
          6000 18.8 sec 24 minutes

          So we restrict our filter and get more efficient depending as we get more detailed and deeper with the query. To toggle between using filter and not using it, I set the hive.optimize.ppd.storage flag to false so no predicate pushdown happens.

          Finally query without M/R job:

          Restrict query by multiple key: (No M/R job)

          Example query: select * from my_table where key.firstpart="something" and key.secondpart="something2";

          Memory(in MB) With filter Without filter
          3000 5 sec 19 minutes
          Show
          swarnim Swarnim Kulkarni added a comment - Here are results from my testing with and without this patch applied. The table "my_table" for this testing contains about 8 M rows. Restrict query by single key : Example query: select * from my_table where key.firstpart="something"; Memory(in MB) With patch Without patch 1500 Out of memory Out of memory 3000 2.5 minutes Out of memory 6000 2.4 minutes 23 minutes Restrict query by multiple key : (Note that the key parts must be successive for this to work) Example query: select * from my_table where key.firstpart="something" and key.secondpart="something2"; Memory(in MB) With filter Without filter 1500 23 sec Out of memory 3000 19 sec Out of memory 6000 18.8 sec 24 minutes So we restrict our filter and get more efficient depending as we get more detailed and deeper with the query. To toggle between using filter and not using it, I set the hive.optimize.ppd.storage flag to false so no predicate pushdown happens. Finally query without M/R job: Restrict query by multiple key : (No M/R job) Example query: select * from my_table where key.firstpart="something" and key.secondpart="something2"; Memory(in MB) With filter Without filter 3000 5 sec 19 minutes
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Uploading patch.

          Show
          swarnim Swarnim Kulkarni added a comment - Uploading patch.

            People

            • Assignee:
              zsombor.klara Barna Zsombor Klara
              Reporter:
              swarnim Swarnim Kulkarni
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development