Hive
  1. Hive
  2. HIVE-5369

Annotate hive operator tree with statistics from metastore

    Details

      Description

      Currently the statistics gathered at table/partition level and column level are not used during query planning stage. Statistics at table/partition and column level can be used for optimizing the query plans. Basic statistics like uncompressed data size can be used for better reducer estimation. Other statistics like number of rows, distinct values of columns, average length of columns etc. can be used by Cost Based Optimizer (CBO) for making better query plan selection. As a first step in improving query planning the statistics that are available in the metastore should be attached to hive operator tree. The operator tree should be walked and annotated with statistics information. The attached statistics will vary for each operator depending on the operation it performs. For example, select operator will change the average row size but doesn't affect the number of rows. Similarly filter operator will change the number of rows but doesn't change the average row size. Similar rules can be applied for other operators as well.

      Rules for different operators are added as comments in the code. For more detailed information, the reference book that I am using is "Database Systems: The Complete Book" by Garcia-Molina et.al.

      1. HIVE-5369.WIP.txt
        146 kB
        Prasanth Jayachandran
      2. HIVE-5369.2.WIP.txt
        874 kB
        Prasanth Jayachandran
      3. HIVE-5369.1.txt
        750 kB
        Prasanth Jayachandran
      4. HIVE-5369.refactor.WIP.txt
        700 kB
        Prasanth Jayachandran
      5. HIVE-5369.2.patch.txt
        725 kB
        Prasanth Jayachandran
      6. HIVE-5369.3.patch.txt
        718 kB
        Prasanth Jayachandran
      7. HIVE-5369.4.patch.txt
        796 kB
        Prasanth Jayachandran
      8. HIVE-5369.5.patch.txt
        800 kB
        Prasanth Jayachandran
      9. HIVE-5369.6.patch.txt
        803 kB
        Prasanth Jayachandran
      10. HIVE-5369.7.patch.txt
        1.23 MB
        Prasanth Jayachandran
      11. HIVE-5369.8.patch.txt
        1.27 MB
        Prasanth Jayachandran
      12. HIVE-5369.9.patch.txt
        1.29 MB
        Prasanth Jayachandran
      13. HIVE-5369.9.patch
        1.29 MB
        Gunther Hagleitner
      14. HIVE-5369.10.patch
        1.29 MB
        Prasanth Jayachandran

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          12d 8h 2m 1 Gunther Hagleitner 18/Nov/13 05:53
          Open Open Patch Available Patch Available
          40d 13h 13m 2 Gunther Hagleitner 18/Nov/13 05:54
          Patch Available Patch Available Resolved Resolved
          13h 36m 1 Harish Butani 18/Nov/13 19:30
          Lefty Leverenz made changes -
          Link This issue is related to HIVE-6300 [ HIVE-6300 ]
          Harish Butani made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Harish Butani added a comment -

          Thank Prasanth. Nice work!

          Show
          Harish Butani added a comment - Thank Prasanth. Nice work!
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.10.patch [ 12614426 ]
          Hide
          Prasanth Jayachandran added a comment -

          Fixed the failing test which was recently added.

          Show
          Prasanth Jayachandran added a comment - Fixed the failing test which was recently added.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12614341/HIVE-5369.9.patch

          ERROR: -1 due to 1 failed/errored test(s), 4617 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/338/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/338/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12614341

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12614341/HIVE-5369.9.patch ERROR: -1 due to 1 failed/errored test(s), 4617 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/338/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/338/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12614341
          Gunther Hagleitner made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Gunther Hagleitner made changes -
          Attachment HIVE-5369.9.patch [ 12614341 ]
          Hide
          Gunther Hagleitner added a comment -

          Re-upload .9 for precommit tests.

          Show
          Gunther Hagleitner added a comment - Re-upload .9 for precommit tests.
          Gunther Hagleitner made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Harish Butani added a comment -

          +1

          Show
          Harish Butani added a comment - +1
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.9.patch.txt [ 12614028 ]
          Hide
          Prasanth Jayachandran added a comment -

          Fixed the 10 failing test cases reported by HIVE QA.

          Show
          Prasanth Jayachandran added a comment - Fixed the 10 failing test cases reported by HIVE QA.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613969/HIVE-5369.8.patch.txt

          ERROR: -1 due to 10 failed/errored test(s), 4619 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/292/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/292/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12613969

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613969/HIVE-5369.8.patch.txt ERROR: -1 due to 10 failed/errored test(s), 4619 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/292/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/292/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12613969
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.8.patch.txt [ 12613969 ]
          Hide
          Prasanth Jayachandran added a comment -

          As per Harish Butani's comment added support for char/varchar. Fixed some more failing tests.

          Show
          Prasanth Jayachandran added a comment - As per Harish Butani 's comment added support for char/varchar. Fixed some more failing tests.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.7.patch.txt [ 12613940 ]
          Hide
          Prasanth Jayachandran added a comment -

          Regenerated golden files for most of the qfiles that contain explain extended.

          Show
          Prasanth Jayachandran added a comment - Regenerated golden files for most of the qfiles that contain explain extended.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.6.patch.txt [ 12613777 ]
          Hide
          Prasanth Jayachandran added a comment -

          Harish Butani Thanks for reviewing the patch. I addressed all your review comments in this patch. Left some comments in RB too. Apart from the code review comments I added the following changes

          1) JOIN rule missed the precondition check. Because of this even if parents column statistics are not available it will try to apply the rule which will result in unexpected exceptions. This is the reason for many test case failures in previous HIVE QA precommit run.
          2) GROUPBY rule used to multiply the number of rows in case of grouping set. So when statistics are updated, column statistics will also be multiplied by a factor. The number of distincts should not be affected because of this multiplier. Added code to protect number of distincts to not change when there is increase in number of rows.
          3) FILTER rule where the predicate is a boolean column or NOT of boolean column is updated to return numTrues and numFalses respectively. This will result in more accurate number of rows than dividing by 2.
          4) Added qfile test to check the effect of map-side parallelism for group by operator.
          5) Removed all PTF related code.

          Still other failing test cases are not fixed. Will look for HIVE QA failure report to regenerate failing tests.

          Show
          Prasanth Jayachandran added a comment - Harish Butani Thanks for reviewing the patch. I addressed all your review comments in this patch. Left some comments in RB too. Apart from the code review comments I added the following changes 1) JOIN rule missed the precondition check. Because of this even if parents column statistics are not available it will try to apply the rule which will result in unexpected exceptions. This is the reason for many test case failures in previous HIVE QA precommit run. 2) GROUPBY rule used to multiply the number of rows in case of grouping set. So when statistics are updated, column statistics will also be multiplied by a factor. The number of distincts should not be affected because of this multiplier. Added code to protect number of distincts to not change when there is increase in number of rows. 3) FILTER rule where the predicate is a boolean column or NOT of boolean column is updated to return numTrues and numFalses respectively. This will result in more accurate number of rows than dividing by 2. 4) Added qfile test to check the effect of map-side parallelism for group by operator. 5) Removed all PTF related code. Still other failing test cases are not fixed. Will look for HIVE QA failure report to regenerate failing tests.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613517/HIVE-5369.5.patch.txt

          ERROR: -1 due to 359 failed/errored test(s), 4614 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_ptf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_position
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_innerjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input39_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_hive_626
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_rc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_vc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_cp
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_noalias
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_macro
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins_mixed
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_no_hooks
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optional_outer
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_multi_insert
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_random
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_push_or
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_null_value
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_exclude_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_router_join_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_user_properties
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt18
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subq_where_serialization
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_exists
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notexists
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case_column_pruning
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_shufflejoin
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_mismatch1
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_sortmerge_mapjoin_mismatch_1
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          org.apache.hive.hcatalog.hbase.TestHiveHBaseTableOutputFormat.org.apache.hive.hcatalog.hbase.TestHiveHBaseTableOutputFormat
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/264/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/264/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 359 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12613517

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613517/HIVE-5369.5.patch.txt ERROR: -1 due to 359 failed/errored test(s), 4614 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_ptf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_position org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_innerjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input39_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_hive_626 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_rc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_vc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_cp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_noalias org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_macro org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins_mixed org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_no_hooks org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optional_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_multi_insert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_random org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_push_or org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_null_value org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_exclude_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_router_join_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_user_properties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subq_where_serialization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_exists org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notexists org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case_column_pruning org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_shufflejoin org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_mismatch1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_sortmerge_mapjoin_mismatch_1 org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.hcatalog.hbase.TestHiveHBaseTableOutputFormat.org.apache.hive.hcatalog.hbase.TestHiveHBaseTableOutputFormat Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/264/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/264/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 359 tests failed This message is automatically generated. ATTACHMENT ID: 12613517
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.5.patch.txt [ 12613517 ]
          Hide
          Prasanth Jayachandran added a comment -

          Added more tests to join rule. Multi-attribute and multi-way join cases added. Made changes to GBY operator where map-side GBY will not apply any rules (assume worst case always).

          Show
          Prasanth Jayachandran added a comment - Added more tests to join rule. Multi-attribute and multi-way join cases added. Made changes to GBY operator where map-side GBY will not apply any rules (assume worst case always).
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.4.patch.txt [ 12613466 ]
          Hide
          Prasanth Jayachandran added a comment -

          Refactored patch. Removed dependency from lineage index. Statistics (along with column statistics) are now passed down the operator tree. Still there are few open issues. Will create sub-tasks for open issues.

          Show
          Prasanth Jayachandran added a comment - Refactored patch. Removed dependency from lineage index. Statistics (along with column statistics) are now passed down the operator tree. Still there are few open issues. Will create sub-tasks for open issues.
          Prasanth Jayachandran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Prasanth Jayachandran added a comment -

          Marking it as patch available to HIVE QA to run tests.

          Show
          Prasanth Jayachandran added a comment - Marking it as patch available to HIVE QA to run tests.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.3.patch.txt [ 12612255 ]
          Hide
          Prasanth Jayachandran added a comment -

          Refreshed patch to the latest trunk (post mavenization).

          Show
          Prasanth Jayachandran added a comment - Refreshed patch to the latest trunk (post mavenization).
          Prasanth Jayachandran made changes -
          Remote Link This issue links to "Review Board Link (Web Link)" [ 13312 ]
          Hide
          Prasanth Jayachandran added a comment -

          RB entry for the new patch https://reviews.apache.org/r/15223/

          Show
          Prasanth Jayachandran added a comment - RB entry for the new patch https://reviews.apache.org/r/15223/
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.2.patch.txt [ 12612081 ]
          Hide
          Prasanth Jayachandran added a comment -

          Added misc fixes and did code cleanup.

          Show
          Prasanth Jayachandran added a comment - Added misc fixes and did code cleanup.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.refactor.WIP.txt [ 12611993 ]
          Hide
          Prasanth Jayachandran added a comment -

          Based on code review comments refactored this patch. It still not complete yet. The patch has to be refreshed with post mavenization trunk. Also code cleanup should be done.

          Show
          Prasanth Jayachandran added a comment - Based on code review comments refactored this patch. It still not complete yet. The patch has to be refreshed with post mavenization trunk. Also code cleanup should be done.
          Hide
          Gunther Hagleitner added a comment -

          Good stuff. Some comments on phabricator.

          Show
          Gunther Hagleitner added a comment - Good stuff. Some comments on phabricator.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.1.txt [ 12607973 ]
          Hide
          Prasanth Jayachandran added a comment -

          Added few more test cases and misc fixes. This patch is now ready for review.

          Show
          Prasanth Jayachandran added a comment - Added few more test cases and misc fixes. This patch is now ready for review.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.2.WIP.txt [ 12607765 ]
          Hide
          Prasanth Jayachandran added a comment -

          Added mergeable interface for statistics merging. Added q file tests for select, filter, group by, join, table stats and part stats. Still some more cleanup and test cases needs to be done.

          Show
          Prasanth Jayachandran added a comment - Added mergeable interface for statistics merging. Added q file tests for select, filter, group by, join, table stats and part stats. Still some more cleanup and test cases needs to be done.
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.txt [ 12607434 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.2.txt [ 12607430 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.2.txt [ 12607432 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.2.txt [ 12607432 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.txt [ 12605224 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-5369.WIP.2.txt [ 12607430 ]
          Hide
          Prasanth Jayachandran added a comment -

          Uploading next version of WIP patch. This new patch adds support for GroupBy, basic Join Operator, fixes for SELECT operator (to support PTF) and other misc fixes. There are still more changes required in Statistics class to support joins on tables from multiple databases.

          Show
          Prasanth Jayachandran added a comment - Uploading next version of WIP patch. This new patch adds support for GroupBy, basic Join Operator, fixes for SELECT operator (to support PTF) and other misc fixes. There are still more changes required in Statistics class to support joins on tables from multiple databases.
          Prasanth Jayachandran made changes -
          Link This issue is blocked by HIVE-5325 [ HIVE-5325 ]
          Hide
          Prasanth Jayachandran added a comment -
          Show
          Prasanth Jayachandran added a comment - Phabricator link https://reviews.facebook.net/D13131
          Prasanth Jayachandran made changes -
          Description Currently the statistics gathered at table/partition level and column level are not used during query planning stage. Statistics at table/partition and column level can be used for optimizing the query plans. Basic statistics like uncompressed data size can be used for better reducer estimation. Other statistics like number of rows, distinct values of columns, average length of columns etc. can be used by Cost Based Optimizer (CBO) for making better query plan selection. As a first step in improving query planning the statistics that are available in the metastore should be attached to hive operator tree. The operator tree should be walked and annotated with statistics information. The attached statistics will vary for each operator depending on the operation it performs. For example, select operator will change the average row size but doesn't affect the number of rows. Similarly filter operator will change the number of rows but doesn't change the average row size. Similar rules can be applied for other operators as well. Currently the statistics gathered at table/partition level and column level are not used during query planning stage. Statistics at table/partition and column level can be used for optimizing the query plans. Basic statistics like uncompressed data size can be used for better reducer estimation. Other statistics like number of rows, distinct values of columns, average length of columns etc. can be used by Cost Based Optimizer (CBO) for making better query plan selection. As a first step in improving query planning the statistics that are available in the metastore should be attached to hive operator tree. The operator tree should be walked and annotated with statistics information. The attached statistics will vary for each operator depending on the operation it performs. For example, select operator will change the average row size but doesn't affect the number of rows. Similarly filter operator will change the number of rows but doesn't change the average row size. Similar rules can be applied for other operators as well.

          Rules for different operators are added as comments in the code. For more detailed information, the reference book that I am using is "Database Systems: The Complete Book" by Garcia-Molina et.al.
          Prasanth Jayachandran made changes -
          Field Original Value New Value
          Attachment HIVE-5369.WIP.txt [ 12605224 ]
          Hide
          Prasanth Jayachandran added a comment -

          Uploading a WIP patch. There are many rough edges which needs to be fixed/addressed.

          Show
          Prasanth Jayachandran added a comment - Uploading a WIP patch. There are many rough edges which needs to be fixed/addressed.
          Prasanth Jayachandran created issue -

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development