Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2923

Lazily register bags with SpillableMemoryManager

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.11
    • None
    • None

    Description

      Currently, all Spillable DataBags get registered by the BagFactory at the moment of creation. In practice, a lot of these bags will not get large enough to be worth spilling; we can avoid a lot of memory overhead and cheapen the process of finding a bag to spill when we do need it, by allowing Bags themselves to register when they grow to some respectable threshold.

      Related JIRAs: PIG-2917, PIG-2918

      Attachments

        1. bagspill_delay.patch
          22 kB
          Dmitriy V. Ryaboy
        2. bagspill_delayed_register.patch
          22 kB
          Dmitriy V. Ryaboy

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. PIG-2963 Illustrate command and POPackageLite

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              dvryaboy Dmitriy V. Ryaboy
              dvryaboy Dmitriy V. Ryaboy
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: