Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-975

Need a databag that does not register with SpillableMemoryManager and spill data pro-actively

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.4.0
    • 0.6.0
    • None
    • None

    Description

      POPackage uses DefaultDataBag during reduce process to hold data. It is registered with SpillableMemoryManager and prone to OutOfMemoryException. It's better to pro-actively managers the usage of the memory. The bag fills in memory to a specified amount, and dump the rest the disk. The amount of memory to hold tuples is configurable. This can avoid out of memory error.

      Attachments

        1. internalbag.xls
          80 kB
          Ying He
        2. PIG-975.patch
          10 kB
          Ying He
        3. PIG-975.patch2
          10 kB
          Ying He
        4. PIG-975.patch3
          10 kB
          Ying He
        5. PIG-975.patch4
          11 kB
          Ying He

        Issue Links

          Activity

            People

              yinghe Ying He
              yinghe Ying He
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: