Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-2803

Severe skew due to null values in columns even when other columns are non-null

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.8.0
    • 0.9.0
    • Functions - Drill
    • None

    Description

      If you have 2 columns that are hashed (either for distribution or for hash based operators) and one of those columns has lots of null values, it can result in substantial skew even if the other column has non-null values.

      In the following query the combined hash value of 2 columns is 0 even when 1 column is non-null. The reason is that if the starting value is null (for cr_reason_sk all values are null in the above query), it does not matter what seed is passed in. The hash function treats the second parameter as a seed and not as a combiner, so it gets ignored.

      select cr_call_center_sk, cr_reason_sk, hash64(cr_reason_sk, hash64(cr_call_center_sk)) as hash_value from catalog_returns  where cr_reason_sk is null and cr_call_center_sk is not null limit 10;
      +-------------------+--------------+------------+
      | cr_call_center_sk | cr_reason_sk | hash_value |
      +-------------------+--------------+------------+
      | 1                 | null         | 0          |
      | 1                 | null         | 0          |
      | 4                 | null         | 0          |
      | 1                 | null         | 0          |
      | 4                 | null         | 0          |
      | 2                 | null         | 0          |
      | 2                 | null         | 0          |
      | 2                 | null         | 0          |
      | 2                 | null         | 0          |
      | 2                 | null         | 0          |
      +-------------------+--------------+------------+
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cchang@maprtech.com Chun Chang
            amansinha100 Aman Sinha
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment