Hive
  1. Hive
  2. HIVE-3381

Result of outer join is not valid

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Outer joins, especially full outer joins or outer join with filter on 'ON clause' is not showing proper results. For example, query in test join_1to1.q

      SELECT * FROM join_1to1_1 a full outer join join_1to1_2 b on a.key1 = b.key1 and a.value = 66 and b.value = 66 ORDER BY a.key1 ASC, a.key2 ASC, a.value ASC, b.key1 ASC, b.key2 ASC, b.value ASC;
      

      results

      NULL	NULL	NULL	NULL	NULL	66
      NULL	NULL	NULL	NULL	10050	66
      NULL	NULL	NULL	10	10010	66
      NULL	NULL	NULL	30	10030	88
      NULL	NULL	NULL	35	10035	88
      NULL	NULL	NULL	40	10040	88
      NULL	NULL	NULL	40	10040	88
      NULL	NULL	NULL	50	10050	88
      NULL	NULL	NULL	50	10050	88
      NULL	NULL	NULL	50	10050	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	66	NULL	NULL	NULL
      NULL	10050	66	NULL	NULL	NULL
      5	10005	66	5	10005	66
      15	10015	66	NULL	NULL	NULL
      20	10020	66	20	10020	66
      25	10025	88	NULL	NULL	NULL
      30	10030	66	NULL	NULL	NULL
      35	10035	88	NULL	NULL	NULL
      40	10040	66	NULL	NULL	NULL
      40	10040	66	40	10040	66
      40	10040	88	NULL	NULL	NULL
      40	10040	88	NULL	NULL	NULL
      50	10050	66	NULL	NULL	NULL
      50	10050	66	50	10050	66
      50	10050	66	50	10050	66
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      70	10040	66	NULL	NULL	NULL
      70	10040	66	NULL	NULL	NULL
      70	10040	66	NULL	NULL	NULL
      70	10040	66	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      

      but it seemed not right. This should be

      NULL	NULL	NULL	NULL	NULL	66
      NULL	NULL	NULL	NULL	10050	66
      NULL	NULL	NULL	10	10010	66
      NULL	NULL	NULL	25	10025	66
      NULL	NULL	NULL	30	10030	88
      NULL	NULL	NULL	35	10035	88
      NULL	NULL	NULL	40	10040	88
      NULL	NULL	NULL	50	10050	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	NULL	70	10040	88
      NULL	NULL	NULL	80	10040	66
      NULL	NULL	NULL	80	10040	66
      NULL	NULL	66	NULL	NULL	NULL
      NULL	10050	66	NULL	NULL	NULL
      5	10005	66	5	10005	66
      15	10015	66	NULL	NULL	NULL
      20	10020	66	20	10020	66
      25	10025	88	NULL	NULL	NULL
      30	10030	66	NULL	NULL	NULL
      35	10035	88	NULL	NULL	NULL
      40	10040	66	40	10040	66
      40	10040	88	NULL	NULL	NULL
      50	10050	66	50	10050	66
      50	10050	66	50	10050	66
      50	10050	88	NULL	NULL	NULL
      50	10050	88	NULL	NULL	NULL
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      60	10040	66	60	10040	66
      70	10040	66	NULL	NULL	NULL
      70	10040	66	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      80	10040	88	NULL	NULL	NULL
      
      1. HIVE-3381.D5565.3.patch
        362 kB
        Phabricator
      2. HIVE-3381.D5565.4.patch
        366 kB
        Phabricator
      3. HIVE-3381.D5565.5.patch
        364 kB
        Phabricator
      4. HIVE-3381.D5565.6.patch
        365 kB
        Phabricator
      5. HIVE-3381.D5565.7.patch
        399 kB
        Phabricator
      6. mapjoin_testOuter.q
        2 kB
        Vikram Dixit K

        Issue Links

          Activity

          Navis created issue -
          Navis made changes -
          Field Original Value New Value
          Link This issue is blocked by HIVE-3411 [ HIVE-3411 ]
          Hide
          Navis added a comment -
          Show
          Navis added a comment - https://reviews.facebook.net/D5565 It was hard.
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Navis added a comment -

          I found a bug with semi join.

          Show
          Navis added a comment - I found a bug with semi join.
          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3381 [jira] Result of outer join is not valid".
          Reviewers: JIRA

          Fixed and rebased on trunk

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          ql/src/test/results/clientpositive/auto_join21.q.out
          ql/src/test/results/clientpositive/auto_join29.q.out
          ql/src/test/results/clientpositive/auto_join7.q.out
          ql/src/test/results/clientpositive/auto_join_filters.q.out
          ql/src/test/results/clientpositive/join21.q.out
          ql/src/test/results/clientpositive/join7.q.out
          ql/src/test/results/clientpositive/join_1to1.q.out
          ql/src/test/results/clientpositive/join_filters.q.out
          ql/src/test/results/clientpositive/join_filters_overlap.q.out

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3381 [jira] Result of outer join is not valid". Reviewers: JIRA Fixed and rebased on trunk REVISION DETAIL https://reviews.facebook.net/D5565 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java ql/src/test/results/clientpositive/auto_join21.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/auto_join7.q.out ql/src/test/results/clientpositive/auto_join_filters.q.out ql/src/test/results/clientpositive/join21.q.out ql/src/test/results/clientpositive/join7.q.out ql/src/test/results/clientpositive/join_1to1.q.out ql/src/test/results/clientpositive/join_filters.q.out ql/src/test/results/clientpositive/join_filters_overlap.q.out To: JIRA, navis
          Phabricator made changes -
          Attachment HIVE-3381.D5565.3.patch [ 12553225 ]
          Hide
          Carl Steinbach added a comment -

          +1

          @Navis: Can you test and commit this patch? Thanks.

          Show
          Carl Steinbach added a comment - +1 @Navis: Can you test and commit this patch? Thanks.
          Hide
          Namit Jain added a comment -

          Navis, can you hold off for a few days ? I also wanted to take a look at this.

          Show
          Namit Jain added a comment - Navis , can you hold off for a few days ? I also wanted to take a look at this.
          Hide
          Ashutosh Chauhan added a comment -

          Namit,
          Since this bug silently results in wrong results, I would like to have this fixed before 0.10 is released. Patch looks good to me. Will wait for your review.

          Show
          Ashutosh Chauhan added a comment - Namit, Since this bug silently results in wrong results, I would like to have this fixed before 0.10 is released. Patch looks good to me. Will wait for your review.
          Hide
          Namit Jain added a comment -

          comments on phabricator

          Show
          Namit Jain added a comment - comments on phabricator
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-3381 [jira] Result of outer join is not valid".

          Navis, I know I am asking pretty naive questions here, but I am not able to proceed.

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:363 Can you add more comments everywhere ?
          I know that the old code also did not have many good comments, but it would be really
          useful.

          It is really difficult for me to be sure.
          I reviewed some test results, and they look fine.

          As you correctly said, this is hard. It would really help if you have a small example
          explained in comments in the code. It would really increase the maintainability of this
          code bigtime. Something like:

          for Query :::
          where the data is :::
          the aliasFilterTags will be::

          I know I am asking for too much, but this code is too risky to change.
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:396 General comment: please add lots of comments for all the private functions.
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:352 What is a skipvector ? Going to the comment below, can you explain with a query/data
          example. Same for offsets.

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-3381 [jira] Result of outer join is not valid". Navis, I know I am asking pretty naive questions here, but I am not able to proceed. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:363 Can you add more comments everywhere ? I know that the old code also did not have many good comments, but it would be really useful. It is really difficult for me to be sure. I reviewed some test results, and they look fine. As you correctly said, this is hard. It would really help if you have a small example explained in comments in the code. It would really increase the maintainability of this code bigtime. Something like: for Query ::: where the data is ::: the aliasFilterTags will be:: I know I am asking for too much, but this code is too risky to change. ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:396 General comment: please add lots of comments for all the private functions. ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:352 What is a skipvector ? Going to the comment below, can you explain with a query/data example. Same for offsets. REVISION DETAIL https://reviews.facebook.net/D5565 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-3381 [jira] Result of outer join is not valid".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:352 'skipVectors' has same meaning with 'inputNulls' in original source, which makes output value for the index(alias) to be filled with null. But doing this I always confused by it's name that it means value for the index(alias) is null, which can be true or false either. I'll change it to 'inputNulls' if you prefer.
          When inputNulls for some index is true, it gets metadata for the index and gets length for it and fills null for that length. 'offsets' is just pre-calculated values of such offsets.
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:363 The problem is I'm also still not sure that this patch is right.
          I'll add more comments.
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:396 ok.

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-3381 [jira] Result of outer join is not valid". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:352 'skipVectors' has same meaning with 'inputNulls' in original source, which makes output value for the index(alias) to be filled with null. But doing this I always confused by it's name that it means value for the index(alias) is null, which can be true or false either. I'll change it to 'inputNulls' if you prefer. When inputNulls for some index is true, it gets metadata for the index and gets length for it and fills null for that length. 'offsets' is just pre-calculated values of such offsets. ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:363 The problem is I'm also still not sure that this patch is right. I'll add more comments. ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:396 ok. REVISION DETAIL https://reviews.facebook.net/D5565 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-3381 [jira] Result of outer join is not valid".

          Navis, I am not saying that the old code is good. On the contrary, it is really difficult to follow.
          There is serious lack of comments in that.
          But, we have to improve that - I really appreciate that you are fixing this very serious bug,
          but it would be really useful if you can add lots of comments so that it becomes much easier to
          maintain/enhance in future.

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-3381 [jira] Result of outer join is not valid". Navis, I am not saying that the old code is good. On the contrary, it is really difficult to follow. There is serious lack of comments in that. But, we have to improve that - I really appreciate that you are fixing this very serious bug, but it would be really useful if you can add lots of comments so that it becomes much easier to maintain/enhance in future. REVISION DETAIL https://reviews.facebook.net/D5565 To: JIRA, navis Cc: njain
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3381 [jira] Result of outer join is not valid".
          Reviewers: JIRA

          1. Simplified code a little
          2. Added comments including example
          3. Rebased on trunk & passed all tests

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          ql/src/test/results/clientpositive/auto_join21.q.out
          ql/src/test/results/clientpositive/auto_join29.q.out
          ql/src/test/results/clientpositive/auto_join7.q.out
          ql/src/test/results/clientpositive/auto_join_filters.q.out
          ql/src/test/results/clientpositive/join21.q.out
          ql/src/test/results/clientpositive/join7.q.out
          ql/src/test/results/clientpositive/join_1to1.q.out
          ql/src/test/results/clientpositive/join_filters.q.out
          ql/src/test/results/clientpositive/join_filters_overlap.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3381 [jira] Result of outer join is not valid". Reviewers: JIRA 1. Simplified code a little 2. Added comments including example 3. Rebased on trunk & passed all tests REVISION DETAIL https://reviews.facebook.net/D5565 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java ql/src/test/results/clientpositive/auto_join21.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/auto_join7.q.out ql/src/test/results/clientpositive/auto_join_filters.q.out ql/src/test/results/clientpositive/join21.q.out ql/src/test/results/clientpositive/join7.q.out ql/src/test/results/clientpositive/join_1to1.q.out ql/src/test/results/clientpositive/join_filters.q.out ql/src/test/results/clientpositive/join_filters_overlap.q.out To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-3381.D5565.4.patch [ 12556036 ]
          Hide
          Ashutosh Chauhan added a comment -

          Namit Jain Navis updated the patch after your comments. Can you take a re-look to see if your comments are addressed. Since this query results in silent failures, we ought to get it fixed soon.

          Show
          Ashutosh Chauhan added a comment - Namit Jain Navis updated the patch after your comments. Can you take a re-look to see if your comments are addressed. Since this query results in silent failures, we ought to get it fixed soon.
          Hide
          Ashutosh Chauhan added a comment -
          Show
          Ashutosh Chauhan added a comment - ping Namit Jain
          Hide
          Vikram Dixit K added a comment -

          Hi Navis,

          Could you kindly refresh this patch as it does not apply cleanly anymore.

          Thanks
          Vikram.

          Show
          Vikram Dixit K added a comment - Hi Navis, Could you kindly refresh this patch as it does not apply cleanly anymore. Thanks Vikram.
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3381 [jira] Result of outer join is not valid".

          Rebased to trunk. running test

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D5565?vs=22947&id=30225#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          ql/src/test/results/clientpositive/auto_join21.q.out
          ql/src/test/results/clientpositive/auto_join29.q.out
          ql/src/test/results/clientpositive/auto_join7.q.out
          ql/src/test/results/clientpositive/auto_join_filters.q.out
          ql/src/test/results/clientpositive/join21.q.out
          ql/src/test/results/clientpositive/join7.q.out
          ql/src/test/results/clientpositive/join_1to1.q.out
          ql/src/test/results/clientpositive/join_filters.q.out
          ql/src/test/results/clientpositive/join_filters_overlap.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3381 [jira] Result of outer join is not valid". Rebased to trunk. running test Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D5565 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D5565?vs=22947&id=30225#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java ql/src/test/results/clientpositive/auto_join21.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/auto_join7.q.out ql/src/test/results/clientpositive/auto_join_filters.q.out ql/src/test/results/clientpositive/join21.q.out ql/src/test/results/clientpositive/join7.q.out ql/src/test/results/clientpositive/join_1to1.q.out ql/src/test/results/clientpositive/join_filters.q.out ql/src/test/results/clientpositive/join_filters_overlap.q.out To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-3381.D5565.5.patch [ 12574749 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3381 [jira] Result of outer join is not valid".

          Fixed test results & passed all tests

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D5565?vs=30225&id=30351#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          ql/src/test/results/clientpositive/auto_join21.q.out
          ql/src/test/results/clientpositive/auto_join29.q.out
          ql/src/test/results/clientpositive/auto_join7.q.out
          ql/src/test/results/clientpositive/auto_join_filters.q.out
          ql/src/test/results/clientpositive/join21.q.out
          ql/src/test/results/clientpositive/join7.q.out
          ql/src/test/results/clientpositive/join_1to1.q.out
          ql/src/test/results/clientpositive/join_filters.q.out
          ql/src/test/results/clientpositive/join_filters_overlap.q.out
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3381 [jira] Result of outer join is not valid". Fixed test results & passed all tests Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D5565 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D5565?vs=30225&id=30351#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java ql/src/test/results/clientpositive/auto_join21.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/auto_join7.q.out ql/src/test/results/clientpositive/auto_join_filters.q.out ql/src/test/results/clientpositive/join21.q.out ql/src/test/results/clientpositive/join7.q.out ql/src/test/results/clientpositive/join_1to1.q.out ql/src/test/results/clientpositive/join_filters.q.out ql/src/test/results/clientpositive/join_filters_overlap.q.out ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-3381.D5565.6.patch [ 12574972 ]
          Hide
          Ashutosh Chauhan added a comment -

          Vikram Dixit K I guess you had some test case. Can you check if thats already covered in patch. If not, can you add that test case here? Also, would you like to verify .q.out changes of patch to make sure they look good to you?

          Show
          Ashutosh Chauhan added a comment - Vikram Dixit K I guess you had some test case. Can you check if thats already covered in patch. If not, can you add that test case here? Also, would you like to verify .q.out changes of patch to make sure they look good to you?
          Hide
          Vikram Dixit K added a comment -

          After going through all of the .q.out changes in this patch, I see that those are the correct results. Great job Navis! Code is more readable and comments are great too.

          +1 (non-binding).

          Show
          Vikram Dixit K added a comment - After going through all of the .q.out changes in this patch, I see that those are the correct results. Great job Navis ! Code is more readable and comments are great too. +1 (non-binding).
          Hide
          Vikram Dixit K added a comment -

          I verified by running against mysql.

          Show
          Vikram Dixit K added a comment - I verified by running against mysql.
          Hide
          Vikram Dixit K added a comment -

          These are some of my tests.

          Show
          Vikram Dixit K added a comment - These are some of my tests.
          Vikram Dixit K made changes -
          Attachment mapjoin_testOuter.q [ 12575374 ]
          Hide
          Ashutosh Chauhan added a comment -

          Navis Can you refresh the patch? Not applying cleanly. I will test and commit it than. Also, it will be good to include vikram's tests while you are refreshing the patch.

          Show
          Ashutosh Chauhan added a comment - Navis Can you refresh the patch? Not applying cleanly. I will test and commit it than. Also, it will be good to include vikram's tests while you are refreshing the patch.
          Hide
          Phabricator added a comment -

          ashutoshc has accepted the revision "HIVE-3381 [jira] Result of outer join is not valid".

          +1 Running tests

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          BRANCH
          DPAL-1739

          ARCANIST PROJECT
          hive

          To: JIRA, ashutoshc, navis
          Cc: njain

          Show
          Phabricator added a comment - ashutoshc has accepted the revision " HIVE-3381 [jira] Result of outer join is not valid". +1 Running tests REVISION DETAIL https://reviews.facebook.net/D5565 BRANCH DPAL-1739 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: njain
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3381 [jira] Result of outer join is not valid".

          Rebased to trunk (HIVE-3980) & added test (Thanks Vikram)

          Reviewers: ashutoshc, JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D5565

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D5565?vs=30351&id=30495#toc

          BRANCH
          DPAL-1739

          ARCANIST PROJECT
          hive

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          ql/src/test/queries/clientpositive/mapjoin_test_outer.q
          ql/src/test/results/clientpositive/auto_join21.q.out
          ql/src/test/results/clientpositive/auto_join29.q.out
          ql/src/test/results/clientpositive/auto_join7.q.out
          ql/src/test/results/clientpositive/auto_join_filters.q.out
          ql/src/test/results/clientpositive/join21.q.out
          ql/src/test/results/clientpositive/join7.q.out
          ql/src/test/results/clientpositive/join_1to1.q.out
          ql/src/test/results/clientpositive/join_filters.q.out
          ql/src/test/results/clientpositive/join_filters_overlap.q.out
          ql/src/test/results/clientpositive/mapjoin1.q.out
          ql/src/test/results/clientpositive/mapjoin_test_outer.q.out

          To: JIRA, ashutoshc, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3381 [jira] Result of outer join is not valid". Rebased to trunk ( HIVE-3980 ) & added test (Thanks Vikram) Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D5565 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D5565?vs=30351&id=30495#toc BRANCH DPAL-1739 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java ql/src/test/queries/clientpositive/mapjoin_test_outer.q ql/src/test/results/clientpositive/auto_join21.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/auto_join7.q.out ql/src/test/results/clientpositive/auto_join_filters.q.out ql/src/test/results/clientpositive/join21.q.out ql/src/test/results/clientpositive/join7.q.out ql/src/test/results/clientpositive/join_1to1.q.out ql/src/test/results/clientpositive/join_filters.q.out ql/src/test/results/clientpositive/join_filters_overlap.q.out ql/src/test/results/clientpositive/mapjoin1.q.out ql/src/test/results/clientpositive/mapjoin_test_outer.q.out To: JIRA, ashutoshc, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-3381.D5565.7.patch [ 12575493 ]
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Navis!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.11.0 [ 12323587 ]
          Resolution Fixed [ 1 ]
          Hide
          Navis added a comment -

          Finally! Thanks to all.

          Show
          Navis added a comment - Finally! Thanks to all.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #2033 (See https://builds.apache.org/job/Hive-trunk-h0.21/2033/)
          HIVE-3381 : Result of outer join is not valid (Navis via Ashutosh Chauhan) (Revision 1461234)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461234
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          • /hive/trunk/ql/src/test/queries/clientpositive/mapjoin_test_outer.q
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join7.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join_filters.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join21.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join7.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_1to1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_filters.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_filters_overlap.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #2033 (See https://builds.apache.org/job/Hive-trunk-h0.21/2033/ ) HIVE-3381 : Result of outer join is not valid (Navis via Ashutosh Chauhan) (Revision 1461234) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461234 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java /hive/trunk/ql/src/test/queries/clientpositive/mapjoin_test_outer.q /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join7.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join_filters.q.out /hive/trunk/ql/src/test/results/clientpositive/join21.q.out /hive/trunk/ql/src/test/results/clientpositive/join7.q.out /hive/trunk/ql/src/test/results/clientpositive/join_1to1.q.out /hive/trunk/ql/src/test/results/clientpositive/join_filters.q.out /hive/trunk/ql/src/test/results/clientpositive/join_filters_overlap.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/)
          HIVE-3381 : Result of outer join is not valid (Navis via Ashutosh Chauhan) (Revision 1461234)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461234
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java
          • /hive/trunk/ql/src/test/queries/clientpositive/mapjoin_test_outer.q
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join7.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/auto_join_filters.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join21.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join7.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_1to1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_filters.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/join_filters_overlap.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/ ) HIVE-3381 : Result of outer join is not valid (Navis via Ashutosh Chauhan) (Revision 1461234) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461234 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java /hive/trunk/ql/src/test/queries/clientpositive/mapjoin_test_outer.q /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join7.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join_filters.q.out /hive/trunk/ql/src/test/results/clientpositive/join21.q.out /hive/trunk/ql/src/test/results/clientpositive/join7.q.out /hive/trunk/ql/src/test/results/clientpositive/join_1to1.q.out /hive/trunk/ql/src/test/results/clientpositive/join_filters.q.out /hive/trunk/ql/src/test/results/clientpositive/join_filters_overlap.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development