Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-975

Need a databag that does not register with SpillableMemoryManager and spill data pro-actively

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      POPackage uses DefaultDataBag during reduce process to hold data. It is registered with SpillableMemoryManager and prone to OutOfMemoryException. It's better to pro-actively managers the usage of the memory. The bag fills in memory to a specified amount, and dump the rest the disk. The amount of memory to hold tuples is configurable. This can avoid out of memory error.

        Attachments

        1. internalbag.xls
          80 kB
          Ying He
        2. PIG-975.patch
          10 kB
          Ying He
        3. PIG-975.patch2
          10 kB
          Ying He
        4. PIG-975.patch3
          10 kB
          Ying He
        5. PIG-975.patch4
          11 kB
          Ying He

          Issue Links

            Activity

              People

              • Assignee:
                yinghe Ying He
                Reporter:
                yinghe Ying He
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: