Pig
  1. Pig
  2. PIG-483

PERFORMANCE: different strategies for large and small order bys

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.2.0
    • Fix Version/s: None
    • Component/s: None

      Description

      Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

      This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

      1. PIG-483.0.patch
        6 kB
        Jie Li
      2. PIG-483.1.patch
        10 kB
        Jie Li

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Jie Li
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development