Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2652

Skew join and order by don't trigger reducer estimation

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Fix how reducers are estimated for skew join and order operators.

      Description

      If neither PARALLEL, default parallel or mapred.reduce.tasks are set, the number of reducers is not estimated based on input size for skew joins or order by. Instead, these jobs get only 1 reducer.

        Attachments

        1. PIG-2652_1.patch
          0.8 kB
          Bill Graham
        2. PIG-2652_2.patch
          2 kB
          Daniel Dai
        3. PIG-2652_3.patch
          10 kB
          Daniel Dai
        4. PIG-2652_3_10.patch
          9 kB
          Daniel Dai
        5. PIG-2652_4.patch
          9 kB
          Dmitriy V. Ryaboy
        6. PIG-2652_5.patch
          11 kB
          Dmitriy V. Ryaboy
        7. PIG-2652_6.patch
          15 kB
          Dmitriy V. Ryaboy
        8. PIG-2652_7.patch
          24 kB
          Dmitriy V. Ryaboy

          Issue Links

            Activity

              People

              • Assignee:
                dvryaboy Dmitriy V. Ryaboy
                Reporter:
                billgraham Bill Graham
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: