Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14797

reducer number estimating may lead to data skew

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Query Processor

    Description

      HiveKey's hash code is generated by multipling by 31 key by key which is implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
      for (int i = 0; i < bucketFields.length; i++)

      { int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], bucketFieldInspectors[i]); hashCode = 31 * hashCode + fieldHash; }

      The follow example will lead to data skew:

      I hava two table called tbl1 and tbl2 and they have the same column: a int, b string. The values of column 'a' in both two tables are not skew, but values of column 'b' in both two tables are skew.

      When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data skew.

      As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the result, the job will be skew.

      Attachments

        1. HIVE-14797.2.patch
          3 kB
          roncenzhao
        2. HIVE-14797.3.patch
          3 kB
          roncenzhao
        3. HIVE-14797.4.patch
          3 kB
          roncenzhao
        4. HIVE-14797.patch
          0.5 kB
          roncenzhao

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            roncenzhao roncenzhao Assign to me
            roncenzhao roncenzhao

            Dates

              Created:
              Updated:

              Slack

                Issue deployment