Pig
  1. Pig
  2. PIG-483

PERFORMANCE: different strategies for large and small order bys

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.2.0
    • Fix Version/s: None
    • Component/s: None

      Description

      Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

      This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

      1. PIG-483.0.patch
        6 kB
        Jie Li
      2. PIG-483.1.patch
        10 kB
        Jie Li

        Issue Links

          Activity

          Olga Natkovich created issue -
          Nigel Daley made changes -
          Field Original Value New Value
          Fix Version/s 1.0.0 [ 12313288 ]
          Nigel Daley made changes -
          Affects Version/s 0.2.0 [ 12313783 ]
          Affects Version/s 1.0.0 [ 12313288 ]
          Daniel Dai made changes -
          Labels gsoc2011
          Description Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer. Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

          This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011
          Dmitriy V. Ryaboy made changes -
          Link This issue is related to PIG-2675 [ PIG-2675 ]
          Jie Li made changes -
          Labels gsoc2011 gsoc2011 performance
          Jie Li made changes -
          Link This issue relates to PIG-2772 [ PIG-2772 ]
          Jie Li made changes -
          Attachment PIG-483.0.patch [ 12533767 ]
          Jie Li made changes -
          Link This issue is blocked by PIG-2652 [ PIG-2652 ]
          Jie Li made changes -
          Link This issue is blocked by PIG-2652 [ PIG-2652 ]
          Jie Li made changes -
          Link This issue is blocked by PIG-2779 [ PIG-2779 ]
          Jie Li made changes -
          Link This issue relates to PIG-2784 [ PIG-2784 ]
          Jie Li made changes -
          Attachment PIG-483.1.patch [ 12538634 ]
          Dmitriy V. Ryaboy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Jie Li [ jay23jack ]
          Bill Graham made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]

            People

            • Assignee:
              Jie Li
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development