Hive
  1. Hive
  2. HIVE-2839

Filters on outer join with mapjoin hint is not applied correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Testing HIVE-2820, I've found some queries with mapjoin hint makes exceptions.

      SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND true limit 10;
      
      FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
      java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
      	at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:363)
      	at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.generateMapJoinOperator(MapJoinProcessor.java:483)
      	at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.transform(MapJoinProcessor.java:689)
      	at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7519)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:891)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
      

      and

      SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND b.key * 10 < '1000' limit 10;
      
      java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:416)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
      	at org.apache.hadoop.mapred.Child.main(Child.java:264)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException
      	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
      	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
      	at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321)
      	at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
      	at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495)
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
      	... 8 more
      
      1. HIVE-2839.D2079.7.patch
        37 kB
        Phabricator
      2. HIVE-2839.D2079.6.patch
        37 kB
        Phabricator
      3. HIVE-2839.D2079.5.patch
        11 kB
        Phabricator
      4. HIVE-2839.D2079.4.patch
        11 kB
        Phabricator
      5. HIVE-2839.D2079.3.patch
        9 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--HIVE-2839.D2079.2.patch
        9 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--HIVE-2839.D2079.1.patch
        9 kB
        Phabricator

        Issue Links

          Activity

          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Phabricator added a comment -

          navis has closed the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, njain, navis
          Cc: njain

          Show
          Phabricator added a comment - navis has closed the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, njain, navis Cc: njain
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #110 (See https://builds.apache.org/job/Hive-trunk-hadoop2/110/)
          HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly
          (Navis via namit) (Revision 1443065)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #110 (See https://builds.apache.org/job/Hive-trunk-hadoop2/110/ ) HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly (Navis via namit) (Revision 1443065) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.11.0 [ 12323587 ]
          Resolution Fixed [ 1 ]
          Hide
          Namit Jain added a comment -

          Committed. Thanks Navis

          Show
          Namit Jain added a comment - Committed. Thanks Navis
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1959 (See https://builds.apache.org/job/Hive-trunk-h0.21/1959/)
          HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly
          (Navis via namit) (Revision 1443065)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1959 (See https://builds.apache.org/job/Hive-trunk-h0.21/1959/ ) HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly (Navis via namit) (Revision 1443065) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Hide
          Hudson added a comment -

          Integrated in hive-trunk-hadoop1 #71 (See https://builds.apache.org/job/hive-trunk-hadoop1/71/)
          HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly
          (Navis via namit) (Revision 1443065)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q
          • /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Show
          Hudson added a comment - Integrated in hive-trunk-hadoop1 #71 (See https://builds.apache.org/job/hive-trunk-hadoop1/71/ ) HIVE-2839 Filters on outer join with mapjoin hint is not applied correctly (Navis via namit) (Revision 1443065) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443065 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java /hive/trunk/ql/src/test/queries/clientpositive/mapjoin1.q /hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out
          Hide
          Phabricator added a comment -

          njain has accepted the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          BRANCH
          DPAL-919

          ARCANIST PROJECT
          hive

          To: JIRA, njain, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has accepted the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". REVISION DETAIL https://reviews.facebook.net/D2079 BRANCH DPAL-919 ARCANIST PROJECT hive To: JIRA, njain, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-2839.D2079.7.patch [ 12568185 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          Addressed comments except removing getSingleParent() method, which is needed for error handling

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D2079?vs=27231&id=27309#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Addressed comments except removing getSingleParent() method, which is needed for error handling Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D2079 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D2079?vs=27231&id=27309#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis Cc: njain
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 You should assert numberParents == 1

          We need to check that before coming to this function.
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 ok - that's fine
          ideally, it should be cleaned up, but can be a follow-up

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 You should assert numberParents == 1 We need to check that before coming to this function. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 ok - that's fine ideally, it should be cleaned up, but can be a follow-up REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 It's just a list of operators which can have multiple parents. The conditions you've mentioned should be checked before calling this. I've done this intentionally cause this is utility class.
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:181 hm.. ok.
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 Return value might be used for setting OperatorDescs, which has ArrayList instead of List. Would it be better to wrap again with ArrayList before setting?

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 It's just a list of operators which can have multiple parents. The conditions you've mentioned should be checked before calling this. I've done this intentionally cause this is utility class. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:181 hm.. ok. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 Return value might be used for setting OperatorDescs, which has ArrayList instead of List. Would it be better to wrap again with ArrayList before setting? REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, navis Cc: njain
          Namit Jain made changes -
          Link This issue is related to HIVE-3988 [ HIVE-3988 ]
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 This should not be possible.

          We don't support

          join -> mapjoin
          union -> mapjoin

          I am not sure about LateralView -> mapjoin,
          but if it is allowed, I will add a jira to stop that, and fix that
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:181 Remove this function.

          assert that parents.size() == 1
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:128 return a List instead of ArrayList
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 result can be a List instead of ArrayList

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:169 This should not be possible. We don't support join -> mapjoin union -> mapjoin I am not sure about LateralView -> mapjoin, but if it is allowed, I will add a jira to stop that, and fix that ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:181 Remove this function. assert that parents.size() == 1 ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:128 return a List instead of ArrayList ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:130 result can be a List instead of ArrayList REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, navis Cc: njain
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Phabricator made changes -
          Attachment HIVE-2839.D2079.6.patch [ 12567919 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          Added tests

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D2079?vs=27177&id=27231#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Added tests Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D2079 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D2079?vs=27177&id=27231#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          INLINE COMMENTS
          ql/src/test/queries/clientpositive/mapjoin1.q:15 Can you add tests with hive.outerjoin.supports.filter also ?

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". INLINE COMMENTS ql/src/test/queries/clientpositive/mapjoin1.q:15 Can you add tests with hive.outerjoin.supports.filter also ? REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-2839.D2079.5.patch [ 12567819 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          Typos, sorry.

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D2079?vs=27171&id=27177#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Typos, sorry. Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D2079 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D2079?vs=27171&id=27177#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis Cc: njain
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.9.0 [ 12317742 ]
          Phabricator made changes -
          Attachment HIVE-2839.D2079.4.patch [ 12567818 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          Addressed comments

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D2079?vs=27147&id=27171#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Addressed comments Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D2079 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D2079?vs=27147&id=27171#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis Cc: njain
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:123 Can you add come comments ?

          Return List instead of ArrayList.
          Same applies to the variables.
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:150 Do you want to add a else ?
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:154 Can you add some comments everywhere ?
          It will make the review easier
          ql/src/test/queries/clientpositive/mapjoin1.q:11 Do you want to add a test with a struct column - and the where clause
          accesses the field

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:123 Can you add come comments ? Return List instead of ArrayList. Same applies to the variables. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:150 Do you want to add a else ? ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:154 Can you add some comments everywhere ? It will make the review easier ql/src/test/queries/clientpositive/mapjoin1.q:11 Do you want to add a test with a struct column - and the where clause accesses the field REVISION DETAIL https://reviews.facebook.net/D2079 To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-2839.D2079.3.patch [ 12567800 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".

          Lost the branch. Recreating.

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D2079?vs=6441&id=27147#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Lost the branch. Recreating. Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D2079 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D2079?vs=6441&id=27147#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out To: JIRA, navis
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          I think we should proceed with this fix.
          It might take some time to completely deprecate this hint.

          Navis, can you refresh ?

          Show
          Namit Jain added a comment - I think we should proceed with this fix. It might take some time to completely deprecate this hint. Navis , can you refresh ?
          Hide
          Ashutosh Chauhan added a comment -

          Now that HIVE-3784 whats the status of this bug? Seems like once we get rid of hint completely, we won't hit this, but I am not sure when will we get there. So feels like this bug still exists. Namit Jain What do you think ? Shall we proceed with this fix?

          Show
          Ashutosh Chauhan added a comment - Now that HIVE-3784 whats the status of this bug? Seems like once we get rid of hint completely, we won't hit this, but I am not sure when will we get there. So feels like this bug still exists. Namit Jain What do you think ? Shall we proceed with this fix?
          Hide
          Navis added a comment -

          I think all of the patches dealing MAPJOIN hint should wait till HIVE-3784 is committed.

          Show
          Navis added a comment - I think all of the patches dealing MAPJOIN hint should wait till HIVE-3784 is committed.
          Hide
          Ashutosh Chauhan added a comment -

          This bug also exists on trunk. Query failed with same stacktrace. Unfortunately cannot resolve conflicts to get the patch applied on trunk for testing.

          Show
          Ashutosh Chauhan added a comment - This bug also exists on trunk. Query failed with same stacktrace. Unfortunately cannot resolve conflicts to get the patch applied on trunk for testing.
          Hide
          Ashutosh Chauhan added a comment -

          This patch got stale as well. Navis, please refresh this patch and I will take a look.

          Show
          Ashutosh Chauhan added a comment - This patch got stale as well. Navis, please refresh this patch and I will take a look.
          Ashutosh Chauhan made changes -
          Fix Version/s 0.9.0 [ 12317742 ]
          Affects Version/s 0.9.0 [ 12317742 ]
          Hide
          Ashutosh Chauhan added a comment -

          Unlinking from 0.9

          Show
          Ashutosh Chauhan added a comment - Unlinking from 0.9
          Phabricator made changes -
          Attachment HIVE-2839.D2079.2.patch [ 12517043 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".
          Reviewers: JIRA

          Fixed rebasing mistakes.

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeUtil.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Reviewers: JIRA Fixed rebasing mistakes. REVISION DETAIL https://reviews.facebook.net/D2079 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeUtil.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Phabricator made changes -
          Field Original Value New Value
          Attachment HIVE-2839.D2079.1.patch [ 12517042 ]
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly".
          Reviewers: JIRA

          DPAL-919 Filters on outer join with mapjoin hint is not applied correctly

          Testing HIVE-2820, I've found some queries with mapjoin hint makes exceptions.

          SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND true limit 10;

          FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
          java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
          at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:363)
          at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.generateMapJoinOperator(MapJoinProcessor.java:483)
          at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.transform(MapJoinProcessor.java:689)
          at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
          at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7519)
          at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
          at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
          at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
          at org.apache.hadoop.hive.ql.Driver.run(Driver.java:891)
          at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
          at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
          at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
          at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
          at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

          and

          SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND b.key * 10 < '1000' limit 10;

          java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException
          at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:416)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
          at org.apache.hadoop.mapred.Child.main(Child.java:264)
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException
          at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
          at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
          at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321)
          at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
          at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
          at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495)
          at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
          ... 8 more

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D2079

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeUtil.java
          ql/src/test/queries/clientpositive/mapjoin1.q
          ql/src/test/results/clientpositive/mapjoin1.q.out

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/4509/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - navis requested code review of " HIVE-2839 [jira] Filters on outer join with mapjoin hint is not applied correctly". Reviewers: JIRA DPAL-919 Filters on outer join with mapjoin hint is not applied correctly Testing HIVE-2820 , I've found some queries with mapjoin hint makes exceptions. SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND true limit 10; FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:363) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.generateMapJoinOperator(MapJoinProcessor.java:483) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.transform(MapJoinProcessor.java:689) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7519) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:891) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) and SELECT /*+ MAPJOIN(a) */ * FROM src a RIGHT OUTER JOIN src b on a.key=b.key AND b.key * 10 < '1000' limit 10; java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2079 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeUtil.java ql/src/test/queries/clientpositive/mapjoin1.q ql/src/test/results/clientpositive/mapjoin1.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4509/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Navis created issue -

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development