Pig
  1. Pig
  2. PIG-3044

Trigger POPartialAgg compaction under GC pressure

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11, 0.10.1
    • Fix Version/s: 0.11, 0.12.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      When pig.exec.mapPartAgg, previously the memory reserved for in-memory aggregation was roughly 1/3 the value of pig.cachedbag.memusage per POPartialAgg operator. Reserving that much RAM could cause problems for jobs that required a lot of memory. POPartialAgg is now spillable, and regardless of the setting of pig.cachedbag.memusage, will happily shrink when memory is needed.
      Show
      When pig.exec.mapPartAgg, previously the memory reserved for in-memory aggregation was roughly 1/3 the value of pig.cachedbag.memusage per POPartialAgg operator. Reserving that much RAM could cause problems for jobs that required a lot of memory. POPartialAgg is now spillable, and regardless of the setting of pig.cachedbag.memusage, will happily shrink when memory is needed.

      Description

      If partial aggregation is turned on in pig 10 and 11, 20% (by default) of the available heap can be consumed by the POPartialAgg operator. This can cause memory issues for jobs that use all, or nearly all, of the heap already.

      If we make POPartialAgg "spillable" (trigger compaction when memory reduction is required), we would be much nicer to high-memory jobs.

      1. PIG-3044-hotfix.patch
        9 kB
        Jonathan Coveney
      2. PIG-3044.2.diff
        7 kB
        Dmitriy V. Ryaboy
      3. PIG-3404.diff
        7 kB
        Dmitriy V. Ryaboy

        Activity

          People

          • Assignee:
            Dmitriy V. Ryaboy
            Reporter:
            Dmitriy V. Ryaboy
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development