Hive
  1. Hive
  2. HIVE-3767

BucketizedHiveInputFormat should be automatically used with Bucketized Map Joins also

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It seems like even after https://issues.apache.org/jira/browse/HIVE-3219 we still need to set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat for bucket map joins (though not SMB joins)

        Issue Links

          Activity

          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3767 BucketizedHiveInputFormat should be automatically used with
          Bucketized Map Joins also (Gang Tim Liu via namit) (Revision 1418286)

          Result = ABORTED
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418286
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
          • /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin13.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin6.q
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3767 BucketizedHiveInputFormat should be automatically used with Bucketized Map Joins also (Gang Tim Liu via namit) (Revision 1418286) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418286 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_1.q /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_2.q /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin13.q /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin6.q
          Ashutosh Chauhan made changes -
          Fix Version/s 0.11 [ 12323587 ]
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1841 (See https://builds.apache.org/job/Hive-trunk-h0.21/1841/)
          HIVE-3767 BucketizedHiveInputFormat should be automatically used with
          Bucketized Map Joins also (Gang Tim Liu via namit) (Revision 1418286)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418286
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
          • /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin13.q
          • /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin6.q
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1841 (See https://builds.apache.org/job/Hive-trunk-h0.21/1841/ ) HIVE-3767 BucketizedHiveInputFormat should be automatically used with Bucketized Map Joins also (Gang Tim Liu via namit) (Revision 1418286) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418286 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_1.q /hive/trunk/ql/src/test/queries/clientpositive/bucket_map_join_2.q /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin13.q /hive/trunk/ql/src/test/queries/clientpositive/bucketmapjoin6.q
          Hide
          Gang Tim Liu added a comment -

          Namit thanks a lot Tim

          Show
          Gang Tim Liu added a comment - Namit thanks a lot Tim
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Namit Jain added a comment -

          Committed. Thanks Tim

          Show
          Namit Jain added a comment - Committed. Thanks Tim
          Hide
          Gang Tim Liu added a comment -

          thanks a lot

          Yes, filed HIVE-3778.

          thanks a lot

          Show
          Gang Tim Liu added a comment - thanks a lot Yes, filed HIVE-3778 . thanks a lot
          Gang Tim Liu made changes -
          Link This issue is related to HIVE-3778 [ HIVE-3778 ]
          Hide
          Namit Jain added a comment -

          +1

          looks good, can you file a follow-up for adding MapJoinDesc.isBucketMapJoin() as part of explain plan ?

          Show
          Namit Jain added a comment - +1 looks good, can you file a follow-up for adding MapJoinDesc.isBucketMapJoin() as part of explain plan ?
          Gang Tim Liu made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          Gang Tim Liu added a comment -

          patch is available.

          Show
          Gang Tim Liu added a comment - patch is available.
          Gang Tim Liu made changes -
          Attachment HIVE-3767.patch.2 [ 12559811 ]
          Show
          Gang Tim Liu added a comment - https://reviews.facebook.net/D7209
          Gang Tim Liu made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Gang Tim Liu made changes -
          Link This issue is related to HIVE-3219 [ HIVE-3219 ]
          Gang Tim Liu made changes -
          Release Note It seems like even after https://issues.apache.org/jira/browse/HIVE-3219 we still need to set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat for bucket map joins (though not SMB joins)
          Description It seems like even after https://issues.apache.org/jira/browse/HIVE-3219 we still need to set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat for bucket map joins (though not SMB joins)
          Gang Tim Liu made changes -
          Component/s Query Processor [ 12312586 ]
          Gang Tim Liu made changes -
          Release Note It seems like even after https://issues.apache.org/jira/browse/HIVE-3219 we still need to set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat for bucket map joins (though not SMB joins)
          Namit Jain made changes -
          Assignee Gang Tim Liu [ gangtimliu ]
          Namit Jain made changes -
          Field Original Value New Value
          Summary BucketizedHiveInputFormat should be automatically used with Bucketized Map Joins alsi BucketizedHiveInputFormat should be automatically used with Bucketized Map Joins also
          Namit Jain created issue -

            People

            • Assignee:
              Gang Tim Liu
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development