Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4422

[Plasma] Enforce memory limit in plasma, rather than relying on dlmalloc_set_footprint_limit

    XMLWordPrintableJSON

    Details

      Description

      Currently, Plasma relies on dlmalloc_set_footprint_limit to limit the memory utilization for Plasma Store. This is restrictive because:

      • It restricts Plasma to dlmalloc, which supports limiting memory footprint, as opposed to other, potentially more performant malloc implementations (e.g., jemalloc)
      • dlmalloc_set_footprint_limit does not guarantee that the limit set by it the amount of usable memory. As such, we might trigger evictions much earlier than hitting this limit, e.g., due to fragmentation or metadata overheads.

      To overcome this, we can impose the memory limit at Plasma by tracking the number of bytes allocated and freed using malloc and free calls. Whenever the allocation reaches the set limit, we fail any subsequent allocations (i.e., return NULL from malloc). This allows Plasma to not be tied to dlmalloc, and also provides more accurate tracking of memory allocation/capacity. 

      Caveat: We will need to make sure that the mmaped files are living on a file system that is a bit larger (depending on malloc implementation) than the Plasma memory limit to account for the extra memory required due to fragmentation/metadata overheads.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                anuragkh Anurag Khandelwal
                Reporter:
                anuragkh Anurag Khandelwal
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m