Pig
  1. Pig
  2. PIG-2652

Skew join and order by don't trigger reducer estimation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Fix how reducers are estimated for skew join and order operators.

      Description

      If neither PARALLEL, default parallel or mapred.reduce.tasks are set, the number of reducers is not estimated based on input size for skew joins or order by. Instead, these jobs get only 1 reducer.

      1. PIG-2652_1.patch
        0.8 kB
        Bill Graham
      2. PIG-2652_2.patch
        2 kB
        Daniel Dai
      3. PIG-2652_3_10.patch
        9 kB
        Daniel Dai
      4. PIG-2652_3.patch
        10 kB
        Daniel Dai
      5. PIG-2652_4.patch
        9 kB
        Dmitriy V. Ryaboy
      6. PIG-2652_5.patch
        11 kB
        Dmitriy V. Ryaboy
      7. PIG-2652_6.patch
        15 kB
        Dmitriy V. Ryaboy
      8. PIG-2652_7.patch
        24 kB
        Dmitriy V. Ryaboy

        Issue Links

          Activity

          Bill Graham made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Rohini Palaniswamy made changes -
          Fix Version/s 0.9.3 [ 12319456 ]
          Fix Version/s 0.10.1 [ 12320547 ]
          Jie Li made changes -
          Link This issue blocks PIG-483 [ PIG-483 ]
          Jie Li made changes -
          Link This issue blocks PIG-483 [ PIG-483 ]
          Jie Li made changes -
          Link This issue relates to PIG-2779 [ PIG-2779 ]
          Dmitriy V. Ryaboy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Daniel Dai made changes -
          Fix Version/s 0.10.1 [ 12320547 ]
          Fix Version/s 0.10.0 [ 12316246 ]
          Dmitriy V. Ryaboy made changes -
          Attachment PIG-2652_7.patch [ 12524406 ]
          Dmitriy V. Ryaboy made changes -
          Assignee Bill Graham [ billgraham ] Dmitriy V. Ryaboy [ dvryaboy ]
          Dmitriy V. Ryaboy made changes -
          Attachment PIG-2652_6.patch [ 12523824 ]
          Dmitriy V. Ryaboy made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Dmitriy V. Ryaboy made changes -
          Attachment PIG-2652_5.patch [ 12523267 ]
          Dmitriy V. Ryaboy made changes -
          Attachment PIG-2652_4.patch [ 12523239 ]
          Daniel Dai made changes -
          Attachment PIG-2652_3_10.patch [ 12522731 ]
          Dmitriy V. Ryaboy made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Daniel Dai made changes -
          Attachment PIG-2652_3.patch [ 12522729 ]
          Daniel Dai made changes -
          Attachment PIG-2652_2.patch [ 12522727 ]
          Dmitriy V. Ryaboy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.10.0 [ 12316246 ]
          Fix Version/s 0.9.3 [ 12319456 ]
          Fix Version/s 0.11 [ 12318878 ]
          Resolution Fixed [ 1 ]
          Bill Graham made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Fix how reducers are estimated for skew join and order operators.
          Assignee Bill Graham [ billgraham ]
          Bill Graham made changes -
          Field Original Value New Value
          Attachment PIG-2652_1.patch [ 12522639 ]
          Bill Graham created issue -

            People

            • Assignee:
              Dmitriy V. Ryaboy
              Reporter:
              Bill Graham
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development