Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-975

Need a databag that does not register with SpillableMemoryManager and spill data pro-actively

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.4.0
    • 0.6.0
    • None
    • None

    Description

      POPackage uses DefaultDataBag during reduce process to hold data. It is registered with SpillableMemoryManager and prone to OutOfMemoryException. It's better to pro-actively managers the usage of the memory. The bag fills in memory to a specified amount, and dump the rest the disk. The amount of memory to hold tuples is configurable. This can avoid out of memory error.

      Attachments

        1. PIG-975.patch4
          11 kB
          Ying He
        2. internalbag.xls
          80 kB
          Ying He
        3. PIG-975.patch3
          10 kB
          Ying He
        4. PIG-975.patch2
          10 kB
          Ying He
        5. PIG-975.patch
          10 kB
          Ying He

        Issue Links

          is a clone of

          Improvement - An improvement or enhancement to an existing feature or task. PIG-636 PERFORMANCE: Use lightweight bag implementations which do not register with SpillableMemoryManager with Combiner

          • Major - Major loss of function.
          • Closed
          is cloned by

          Improvement - An improvement or enhancement to an existing feature or task. PIG-1000 InternalCachedBag.java generates javac warning and findbug warning

          • Major - Major loss of function.
          • Closed

          Activity

            People

              yinghe Ying He
              yinghe Ying He
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: