Hive
  1. Hive
  2. HIVE-5056

MapJoinProcessor ignores order of values in removing RS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Query Processor
    • Labels:
      None
    1. HIVE-5056.D12147.3.patch
      34 kB
      Phabricator
    2. HIVE-5056.D12147.2.patch
      34 kB
      Phabricator
    3. HIVE-5056.D12147.1.patch
      33 kB
      Phabricator

      Issue Links

        Activity

        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS".

        Reviewers: JIRA

        HIVE-5056 MapJoinProcessor ignores order of values in removing RS

        http://www.mail-archive.com/user@hive.apache.org/msg09073.html

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D12147

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        ql/src/test/results/clientpositive/auto_join_reordering_values.q.out

        MANAGE HERALD RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/29031/

        To: JIRA, navis

        Show
        Phabricator added a comment - navis requested code review of " HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS". Reviewers: JIRA HIVE-5056 MapJoinProcessor ignores order of values in removing RS http://www.mail-archive.com/user@hive.apache.org/msg09073.html TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D12147 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/test/queries/clientpositive/auto_join_reordering_values.q ql/src/test/results/clientpositive/auto_join_reordering_values.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/29031/ To: JIRA, navis
        Hide
        Chun Chen added a comment -

        Hi, Navis. I and Zhichun Wu finally find the problem after a week of seeking. Since "deal".hashCode()%16=12, "dim_pay_date".hashCode()%16=12, in SemanticAnalyzer#genJoinOperatorChildren, we put "deal", "dim_pay_date" in hash map in sequence, however in MapJoinProcessor#convertMapJoin, we put "dim_pay_date", "deal" in hash map in sequence. And because of hash collision, the traversal order is related to the order of elements putting in the hashmap and result in bug.

        Show
        Chun Chen added a comment - Hi, Navis . I and Zhichun Wu finally find the problem after a week of seeking. Since "deal".hashCode()%16=12, "dim_pay_date".hashCode()%16=12, in SemanticAnalyzer#genJoinOperatorChildren, we put "deal", "dim_pay_date" in hash map in sequence, however in MapJoinProcessor#convertMapJoin, we put "dim_pay_date", "deal" in hash map in sequence. And because of hash collision, the traversal order is related to the order of elements putting in the hashmap and result in bug.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12597334/HIVE-5056.D12147.1.patch

        ERROR: -1 due to 55 failed/errored test(s), 2777 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_smb_bucketmapjoin
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_sortmerge_mapjoin_mismatch_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_empty
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_mismatch1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/390/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/390/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 55 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12597334/HIVE-5056.D12147.1.patch ERROR: -1 due to 55 failed/errored test(s), 2777 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_smb_bucketmapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_sortmerge_mapjoin_mismatch_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_empty org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_mismatch1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/390/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/390/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 55 tests failed This message is automatically generated.
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS".

        Fix tests

        Reviewers: JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12147

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12147?vs=37557&id=37563#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        ql/src/test/results/clientpositive/auto_join_reordering_values.q.out

        To: JIRA, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS". Fix tests Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D12147 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12147?vs=37557&id=37563#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/test/queries/clientpositive/auto_join_reordering_values.q ql/src/test/results/clientpositive/auto_join_reordering_values.q.out To: JIRA, navis
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12597389/HIVE-5056.D12147.2.patch

        SUCCESS: +1 2777 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/396/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/396/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12597389/HIVE-5056.D12147.2.patch SUCCESS: +1 2777 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/396/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/396/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Xuefu Zhang added a comment -

        Could anyone give concise description with enough details for other people to understand the bug? Abbreviation sometimes cause confusion too. RS? Sorry if this is obvious.

        Show
        Xuefu Zhang added a comment - Could anyone give concise description with enough details for other people to understand the bug? Abbreviation sometimes cause confusion too. RS? Sorry if this is obvious.
        Hide
        Ashutosh Chauhan added a comment -

        Navis As requested, would you like to describe the bug and subsequent fix in some detail?

        Show
        Ashutosh Chauhan added a comment - Navis As requested, would you like to describe the bug and subsequent fix in some detail?
        Hide
        Navis added a comment -
        RS := ReduceSinkOperator, JOIN := JoinOperator, RR := RowResolver
        
        If there are 0:T1(a1,a2) and 1:T2(b1,b2) and JOIN has value expression like {0:[T1.a1,T2.a2], 1:[T2.b2,T2.b1]}, output of JOIN is like this.
        
        output : expression 
         _col0 : T1.a1
         _col1 : T1.a2
         _col2 : T2.b2 (not b1)
         _col3 : T2.b1 (not b2)
        

        MapJoinProcessor should remove RSs, which means column expressions(ExprNodeDescs) are not valid and should be rebased on parent of RSs (because expressions should be based on RR of parent). Previously, hive found columns by referencing RR of parents of RSs. In this case, the order is something like [T1.a1-T2.a2-T2.b1-T2.b2], which results invalid column mapping like this.

        output : expression 
         _col0 : T1.a1
         _col1 : T1.a2
         _col2 : T2.b1
         _col3 : T2.b2
        
        Show
        Navis added a comment - RS := ReduceSinkOperator, JOIN := JoinOperator, RR := RowResolver If there are 0:T1(a1,a2) and 1:T2(b1,b2) and JOIN has value expression like {0:[T1.a1,T2.a2], 1:[T2.b2,T2.b1]}, output of JOIN is like this. output : expression _col0 : T1.a1 _col1 : T1.a2 _col2 : T2.b2 (not b1) _col3 : T2.b1 (not b2) MapJoinProcessor should remove RSs, which means column expressions(ExprNodeDescs) are not valid and should be rebased on parent of RSs (because expressions should be based on RR of parent). Previously, hive found columns by referencing RR of parents of RSs. In this case, the order is something like [T1.a1-T2.a2-T2.b1-T2.b2] , which results invalid column mapping like this. output : expression _col0 : T1.a1 _col1 : T1.a2 _col2 : T2.b1 _col3 : T2.b2
        Hide
        Ashutosh Chauhan added a comment -

        should be rebased on parent of RSs (because expressions should be based on RR of parent). Previously, hive found columns by referencing RR of parents of RSs.

        Confused about above. You meant, previously hive found columns by referencing RR of RSs? Not clear about before and after scenario.

        Show
        Ashutosh Chauhan added a comment - should be rebased on parent of RSs (because expressions should be based on RR of parent). Previously, hive found columns by referencing RR of parents of RSs. Confused about above. You meant, previously hive found columns by referencing RR of RSs? Not clear about before and after scenario.
        Hide
        Navis added a comment -
        TS1 TS2
        RS1 RS2
         JOIN
        
        TS1=a1:a2
        TS2=b1:b2
        JOIN=T1.a1:T2.a2:T2.b2:T2.b1
        

        For MapJoin, RS1/RS2 should be removed here. Expressions of JOIN (T1.a1:T2.a2:T2.b2:T2.b1) based on RS1/RS2 should be rebased on TS1/TS2.

        Before the patch, expressions for JOIN is remade by iterating columns of TS1/TS2 and old expressions are just used for checking existence, like this

        for (pos = 0; pos < newParentOps.size(); pos++) {
            Map<String, ExprNodeDesc> colExprMap = op.getColumnExprMap();
            for (Map.Entry<Byte, List<ExprNodeDesc>> entry : valueExprs.entrySet()) {
        

        And that may change the order. In this case, "T1.a1:T2.a2:T2.b2:T2.b1" is changed to "T1.a1:T2.a2:T2.b1:T2.b2"

        Show
        Navis added a comment - TS1 TS2 RS1 RS2 JOIN TS1=a1:a2 TS2=b1:b2 JOIN=T1.a1:T2.a2:T2.b2:T2.b1 For MapJoin, RS1/RS2 should be removed here. Expressions of JOIN (T1.a1:T2.a2:T2.b2:T2.b1) based on RS1/RS2 should be rebased on TS1/TS2. Before the patch, expressions for JOIN is remade by iterating columns of TS1/TS2 and old expressions are just used for checking existence, like this for (pos = 0; pos < newParentOps.size(); pos++) { Map<String, ExprNodeDesc> colExprMap = op.getColumnExprMap(); for (Map.Entry<Byte, List<ExprNodeDesc>> entry : valueExprs.entrySet()) { And that may change the order. In this case, "T1.a1:T2.a2:T2.b2:T2.b1" is changed to "T1.a1:T2.a2:T2.b1:T2.b2"
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS".

        Rebased to trunk

        Reviewers: JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12147

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12147?vs=37563&id=39747#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        ql/src/test/results/clientpositive/auto_join_reordering_values.q.out

        To: JIRA, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS". Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D12147 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12147?vs=37563&id=39747#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/test/queries/clientpositive/auto_join_reordering_values.q ql/src/test/results/clientpositive/auto_join_reordering_values.q.out To: JIRA, navis
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12602093/HIVE-5056.D12147.3.patch

        ERROR: -1 due to 4 failed/errored test(s), 3087 tests executed
        Failed tests:

        org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag
        org.apache.hive.hcatalog.mapreduce.TestHCatExternalHCatNonPartitioned.testHCatNonPartitionedTable
        org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
        org.apache.hive.hcatalog.pig.TestHCatStorer.testPartColsInData
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/667/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/667/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 4 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12602093/HIVE-5056.D12147.3.patch ERROR: -1 due to 4 failed/errored test(s), 3087 tests executed Failed tests: org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag org.apache.hive.hcatalog.mapreduce.TestHCatExternalHCatNonPartitioned.testHCatNonPartitionedTable org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask org.apache.hive.hcatalog.pig.TestHCatStorer.testPartColsInData Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/667/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/667/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 4 tests failed This message is automatically generated.
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS".

        +1

        REVISION DETAIL
        https://reviews.facebook.net/D12147

        BRANCH
        HIVE-5056

        ARCANIST PROJECT
        hive

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-5056 [jira] MapJoinProcessor ignores order of values in removing RS". +1 REVISION DETAIL https://reviews.facebook.net/D12147 BRANCH HIVE-5056 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Navis!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #92 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/92/)
        HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        • /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #92 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/92/ ) HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop1-ptest #160 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/160/)
        HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        • /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #160 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/160/ ) HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-h0.21 #2323 (See https://builds.apache.org/job/Hive-trunk-h0.21/2323/)
        HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        • /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-h0.21 #2323 (See https://builds.apache.org/job/Hive-trunk-h0.21/2323/ ) HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-hadoop2 #420 (See https://builds.apache.org/job/Hive-trunk-hadoop2/420/)
        HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
        • /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #420 (See https://builds.apache.org/job/Hive-trunk-hadoop2/420/ ) HIVE-5056 : MapJoinProcessor ignores order of values in removing RS (Navis Ryu via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1521552 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/test/queries/clientpositive/auto_join_reordering_values.q /hive/trunk/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
        Hide
        Brock Noland added a comment -

        Since this is a bug I think we should commit to 0.12. Any objections?

        Show
        Brock Noland added a comment - Since this is a bug I think we should commit to 0.12. Any objections?
        Hide
        Ashutosh Chauhan added a comment -

        I will let Thejas M Nair decide that since he is the release manager for 0.12 Thejas, you OK with this?

        Show
        Ashutosh Chauhan added a comment - I will let Thejas M Nair decide that since he is the release manager for 0.12 Thejas, you OK with this?
        Hide
        Thejas M Nair added a comment -

        Yes, it would be good to have in 0.12 . I will commit it.

        Show
        Thejas M Nair added a comment - Yes, it would be good to have in 0.12 . I will commit it.
        Hide
        Thejas M Nair added a comment -

        Patch committed to 0.12 branch as well.

        Show
        Thejas M Nair added a comment - Patch committed to 0.12 branch as well.
        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development