Hive
  1. Hive
  2. HIVE-7469

skew join keys when two join table have the same big skew key

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key , the key point is that use mapjoin to deal with skew key, but there has a case which troubles me:
      if the two join tables have the same big skew key on one value :
      for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
      so ,how to fix this case? Will it rollback to common join ? Thanks.

        Activity

        WangMeng created issue -
        WangMeng made changes -
        Field Original Value New Value
        Description In https://issues.apache.org/jira/browse/HIVE-964, I have an general idea about how to deal with skew join key ,but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.
        In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key ,but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.
        WangMeng made changes -
        Description In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key ,but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.
        In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key , the key is that use mapjoin to deal with skew key, but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.
        WangMeng made changes -
        Description In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key , the key is that use mapjoin to deal with skew key, but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.
        In https://issues.apache.org/jira/browse/HIVE-964, I have a general idea about how to deal with skew join key , the key point is that use mapjoin to deal with skew key, but there has a case which troubles me:
        if the two join tables have the same big skew key on one value :
        for example , select * from table A join B on A.id=b.id, both table A and B have a lot of keys on id=1, in this case , if we use map join to deal with the skew key id=1 ,maybe it will OOM.
        so ,how to fix this case? Will it rollback to common join ? Thanks.

          People

          • Assignee:
            Unassigned
            Reporter:
            WangMeng
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development