Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-483

PERFORMANCE: different strategies for large and small order bys

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.2.0
    • Fix Version/s: None
    • Component/s: None

      Description

      Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

      This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

        Attachments

        1. PIG-483.0.patch
          6 kB
          Jie Li
        2. PIG-483.1.patch
          10 kB
          Jie Li

          Issue Links

            Activity

              People

              • Assignee:
                jay23jack Jie Li
                Reporter:
                olgan Olga Natkovich
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: