Hive
  1. Hive
  2. HIVE-6883

Dynamic partitioning optimization does not honor sort order or order by

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0, 0.13.1
    • Component/s: None
    • Labels:
      None

      Description

      HIVE-6455 patch does not honor sort order of the output table or order by of select statement. The reason for the former is numDistributionKey in ReduceSinkDesc is set wrongly. It doesn't take into account the sort columns, because of this RSOp sets the sort columns to null in Key. Since nulls are set in place of sort columns in Key, the sort columns in Value are not sorted.

      The other issue is ORDER BY columns are not honored during insertion. For example

      insert overwrite table over1k_part_orc partition(ds="foo", t) select si,i,b,f,t from over1k_orc where t is null or t=27 order by si;
      

      the select query performs order by on column 'si' in the first MR job. The following MR job (inserted by HIVE-6455), sorts the input data on dynamic partition column 't' without taking into account the already sorted 'si' column. This results in out of order insertion for 'si' column.

      1. HIVE-6883.1.patch
        1.46 MB
        Prasanth Jayachandran
      2. HIVE-6883.2.patch
        1.48 MB
        Prasanth Jayachandran
      3. HIVE-6883.3.patch
        1.48 MB
        Prasanth Jayachandran
      4. HIVE-6883-branch-0.13.3.patch
        1.79 MB
        Prasanth Jayachandran

        Issue Links

          Activity

          Prasanth Jayachandran created issue -
          Prasanth Jayachandran made changes -
          Field Original Value New Value
          Fix Version/s 0.13.0 [ 12324986 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-6883.1.patch [ 12639552 ]
          Prasanth Jayachandran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Prasanth Jayachandran made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 14840 ]
          Prasanth Jayachandran made changes -
          Fix Version/s 0.14.0 [ 12326450 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Prasanth Jayachandran made changes -
          Fix Version/s 0.14.0 [ 12326450 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-6883.2.patch [ 12639645 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-6883.3.patch [ 12639717 ]
          Prasanth Jayachandran made changes -
          Fix Version/s 0.13.0 [ 12324986 ]
          Sergey Shelukhin made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.14.0 [ 12326450 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Prasanth Jayachandran made changes -
          Attachment HIVE-6883-branch-0.13.3.patch [ 12643461 ]
          Sushanth Sowmyan made changes -
          Fix Version/s 0.13.1 [ 12326829 ]
          Thejas M Nair made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Prasanth Jayachandran made changes -
          Link This issue relates to HIVE-8162 [ HIVE-8162 ]

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development