[ARROW-4422] [Plasma] Enforce memory limit in plasma, rather than relying on dlmalloc_set_footprint_limit

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0
• Component/s: C++, C++ - Plasma
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/15957
• Assignee: Anurag Khandelwal
• Reporter: Anurag Khandelwal
• Created: 30/Jan/19 00:53
• Updated: 11/Jan/23 07:33
• Resolved: 31/Jan/19 01:45

Description

Currently, Plasma relies on dlmalloc_set_footprint_limit to limit the memory utilization for Plasma Store. This is restrictive because:

• It restricts Plasma to dlmalloc, which supports limiting memory footprint, as opposed to other, potentially more performant malloc implementations (e.g., jemalloc)
• dlmalloc_set_footprint_limit does not guarantee that the limit set by it the amount of usable memory. As such, we might trigger evictions much earlier than hitting this limit, e.g., due to fragmentation or metadata overheads.

To overcome this, we can impose the memory limit at Plasma by tracking the number of bytes allocated and freed using malloc and free calls. Whenever the allocation reaches the set limit, we fail any subsequent allocations (i.e., return NULL from malloc). This allows Plasma to not be tied to dlmalloc, and also provides more accurate tracking of memory allocation/capacity. 

Caveat: We will need to make sure that the mmaped files are living on a file system that is a bit larger (depending on malloc implementation) than the Plasma memory limit to account for the extra memory required due to fragmentation/metadata overheads.

Issue Links

• links to: GitHub Pull Request #3526