Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      I've gotten feedback over time saying that it's very difficult to grok our explain command. There's supposedly a lot of information that mainly matters to developers or the testing framework. Comparing it to other major DBs it does seem like we're packing way more into explain than other folks.

      I've gone through the explain checking, what could be done to improve readability. Here's a list of things I've found:

      • AST (unreadable in it's "lisp" syntax, not really required for end users)
      • Vectorization (enough to display once per task and only when true)
      • Expressions representation is very lengthy, could be much more compact
      • "if not exists" on DDL (enough to display only on true, or maybe not at all)
      • bucketing info (enough if displayed only if table is actually bucketed)
      • external flag (show only if external)
      • GlobalTableId (don't need in plain explain, maybe in extended)
      • Position of big table (already clear from plan)
      • Stats always (Most DBs mostly only show stats in explain, that gives a sense of what the planer thinks will happen)
      • skew join (only if true should be enough)
      • limit doesn't show the actual limit
      • "Alias -> Map Operator tree" -> alias is duplicated in TableScan operator
      • tag is only useful at runtime (move to explain extended)
      • Some names are camel case or abbreviated, clearer if full name
      • Tez is missing vertex map (aka edges)
      • explain formatted (json) is broken right now (swallows some information)

      Since changing explain results in many golden file updates, i'd like to take a stab at all of these at once.

      1. HIVE-6267.1.partial
        33 kB
        Gunther Hagleitner
      2. HIVE-6267.2.partial
        33 kB
        Gunther Hagleitner
      3. HIVE-6267.3.partial
        33 kB
        Gunther Hagleitner
      4. HIVE-6267.4.patch
        9.80 MB
        Gunther Hagleitner
      5. HIVE-6267.5.patch
        9.88 MB
        Gunther Hagleitner
      6. HIVE-6267.6.patch
        9.91 MB
        Gunther Hagleitner
      7. HIVE-6267.7.patch.gz
        607 kB
        Gunther Hagleitner
      8. HIVE-6267.8.patch
        0.6 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          Here's a draft. Still have to run and evaluate golden files.

          Show
          Gunther Hagleitner added a comment - Here's a draft. Still have to run and evaluate golden files.
          Hide
          Gunther Hagleitner added a comment -

          Oops. Wrong file. 6267.1 is the right one.

          Show
          Gunther Hagleitner added a comment - Oops. Wrong file. 6267.1 is the right one.
          Hide
          Gunther Hagleitner added a comment -

          .2 fixes bug in previous patch. Overriding toStringTree for ASTNodes to change the explain output was not such a hot idea (used for row resolvers).

          Show
          Gunther Hagleitner added a comment - .2 fixes bug in previous patch. Overriding toStringTree for ASTNodes to change the explain output was not such a hot idea (used for row resolvers).
          Hide
          Gunther Hagleitner added a comment -

          Fixed another issue. Removing any duplicates from being printed in explain is too aggressive for some things.

          Show
          Gunther Hagleitner added a comment - Fixed another issue. Removing any duplicates from being printed in explain is too aggressive for some things.
          Show
          Gunther Hagleitner added a comment - https://reviews.apache.org/r/17597/
          Hide
          Vikram Dixit K added a comment -

          LGTM +1 (Pending tests run).

          Show
          Vikram Dixit K added a comment - LGTM +1 (Pending tests run).
          Hide
          Gunther Hagleitner added a comment -

          .4 contains all the changed golden files.

          Show
          Gunther Hagleitner added a comment - .4 contains all the changed golden files.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626526/HIVE-6267.4.patch

          ERROR: -1 due to 36 failed/errored test(s), 4997 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1152/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1152/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 36 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12626526

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626526/HIVE-6267.4.patch ERROR: -1 due to 36 failed/errored test(s), 4997 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1152/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1152/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 36 tests failed This message is automatically generated. ATTACHMENT ID: 12626526
          Hide
          Gunther Hagleitner added a comment -

          some fixes in .5

          Show
          Gunther Hagleitner added a comment - some fixes in .5
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626529/HIVE-6267.5.patch

          ERROR: -1 due to 24 failed/errored test(s), 4997 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1153/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1153/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 24 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12626529

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626529/HIVE-6267.5.patch ERROR: -1 due to 24 failed/errored test(s), 4997 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1153/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1153/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed This message is automatically generated. ATTACHMENT ID: 12626529
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626566/HIVE-6267.6.patch

          ERROR: -1 due to 103 failed/errored test(s), 4997 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_hive_626
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_rc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_vc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1154/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1154/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 103 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12626566

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626566/HIVE-6267.6.patch ERROR: -1 due to 103 failed/errored test(s), 4997 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_hive_626 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_rc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_vc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1 org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1154/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1154/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 103 tests failed This message is automatically generated. ATTACHMENT ID: 12626566
          Hide
          Gunther Hagleitner added a comment -

          Tests look good now. I've run different subsets of the golden files (upload limit) to prove that everything is looking good. There's 3 index tests that fail because their stats are unreliable. I will take out the "explains" temporarily and fix in different jira. This is basically exposing a problem - not something that was introduced in this patch.

          Show
          Gunther Hagleitner added a comment - Tests look good now. I've run different subsets of the golden files (upload limit) to prove that everything is looking good. There's 3 index tests that fail because their stats are unreliable. I will take out the "explains" temporarily and fix in different jira. This is basically exposing a problem - not something that was introduced in this patch.
          Hide
          Gunther Hagleitner added a comment -

          By "unreliable" I mean that they produce different results when run multiple times in sequence in terms of data size of the stats.

          Show
          Gunther Hagleitner added a comment - By "unreliable" I mean that they produce different results when run multiple times in sequence in terms of data size of the stats.
          Hide
          Gunther Hagleitner added a comment -

          .7 has all changes (compressed)

          Show
          Gunther Hagleitner added a comment - .7 has all changes (compressed)
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Thanks for the review Vikram!

          I'm keeping this open and kick off another test run to make sure everything is going right, since this changes a lot of files.

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Thanks for the review Vikram! I'm keeping this open and kick off another test run to make sure everything is going right, since this changes a lot of files.
          Hide
          Gunther Hagleitner added a comment -

          .8 is just to kick a test run and verify.

          Show
          Gunther Hagleitner added a comment - .8 is just to kick a test run and verify.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626583/HIVE-6267.8.patch

          SUCCESS: +1 4997 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12626583

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626583/HIVE-6267.8.patch SUCCESS: +1 4997 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12626583
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Test ran: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/

          No failures.

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Test ran: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1157/ No failures.
          Hide
          Navis added a comment -

          It's breaking all of pending patches. Is this that good?

          Show
          Navis added a comment - It's breaking all of pending patches. Is this that good?
          Hide
          Gunther Hagleitner added a comment -

          Sorry Navis. I tried to keep the disruption as minimal as possible. I collected all things I thought need fixing together before making the changes. Then, I waited until the weekend and a time when the queue is empty and tried to get everything back in shape before people start working again.

          I think it's worth it, otherwise I wouldn't have spent so much time on it. As I mentioned above, I've gotten feedback multiple times about seeing if we can improve explain. Unfortunately, that means tons of golden files. If you can think of a better way I can back out and try again. But it's not clear to me how to avoid changing that many golden files, since we rely on it so heavily in the q files..

          Show
          Gunther Hagleitner added a comment - Sorry Navis . I tried to keep the disruption as minimal as possible. I collected all things I thought need fixing together before making the changes. Then, I waited until the weekend and a time when the queue is empty and tried to get everything back in shape before people start working again. I think it's worth it, otherwise I wouldn't have spent so much time on it. As I mentioned above, I've gotten feedback multiple times about seeing if we can improve explain. Unfortunately, that means tons of golden files. If you can think of a better way I can back out and try again. But it's not clear to me how to avoid changing that many golden files, since we rely on it so heavily in the q files..
          Hide
          Navis added a comment -

          Gunther Hagleitner Generally looks good to me. Simple and concise but things like "Position of Big Table" should not be removed, imho. I think I was a little pissed off yesterday, which was Sunday for you but Monday for me. I apologize for the rudeness.

          Show
          Navis added a comment - Gunther Hagleitner Generally looks good to me. Simple and concise but things like "Position of Big Table" should not be removed, imho. I think I was a little pissed off yesterday, which was Sunday for you but Monday for me. I apologize for the rudeness.

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development