Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2888

Improve performance of POPartialAgg

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.11
    • None
    • None

    Description

      During performance testing, we found that POPartialAgg can cause performance degradation for Pig jobs when the Algebraic UDFs it's being applied to aren't well suited to the operator's assumptions. Changing the implementation to a more flexible hash-based model can provide significant performance improvements.

      Attachments

        1. PIG-2888.final.patch
          97 kB
          Dmitriy V. Ryaboy
        2. partialagg_patch_6.patch
          44 kB
          Dmitriy V. Ryaboy
        3. partialagg_patch_5.patch
          43 kB
          Dmitriy V. Ryaboy
        4. partialagg_patch_4.patch
          42 kB
          Dmitriy V. Ryaboy
        5. partialagg_patch_3.patch
          40 kB
          Dmitriy V. Ryaboy
        6. partialagg_patch_2.patch
          39 kB
          Dmitriy V. Ryaboy
        7. partialagg_patch_1.patch
          29 kB
          Dmitriy V. Ryaboy

        Activity

          People

            dvryaboy Dmitriy V. Ryaboy
            dvryaboy Dmitriy V. Ryaboy
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: