Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1815

pig task retains used instances of PhysicalPlan

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None

      Description

      map tasks of a pig query ran out of memory because there were too many (thousands) instances of combiner PhysicalPlan in memory. Each physical plan (except the last?) was linked to older one as shown in the yourkit snapshot that I am attaching.

      This problem was noticed with 0.8 because of the split combination feature, that resulted in each map having larger inputs. The query also had large physical plan because of multi-query, it had 17 MR jobs merged into one during the multi-query optimization phase.

        Attachments

        1. PIG-1815.1.patch
          1.0 kB
          Thejas M Nair
        2. yourkit_combiner_hprof.jpg
          188 kB
          Thejas M Nair

          Issue Links

            Activity

              People

              • Assignee:
                thejas Thejas M Nair
                Reporter:
                thejas Thejas M Nair
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: