[ARROW-2879] [Python] Arrow plasma can only use a small part of specified shared memory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/19250
• Assignee: Unassigned
• Reporter: chineking
• Created: 19/Jul/18 03:34
• Updated: 11/Jan/23 07:23
• Resolved: 19/Feb/21 21:00

Description

Hi, thanks for the great job of arrow, it helps us a lot.

However, we encounter a problem when we were using plasma.

The sample code:

import numpy as np
import pyarrow as pa
import pyarrow.plasma as plasma

client = plasma.connect("/tmp/plasma", "", 0)

puts = []
nbytes = 0
while True:
    a = np.ones((1000, 1000))
    try:
        oid = client.put(a)
        puts.append(client.get(oid))
        nbytes += a.nbytes
    except pa.lib.PlasmaStoreFull:
        print('use nbytes', nbytes)
        break

We start a plasma store with 1G memory, but the nbytes output above is only 496000000, which cannot even reach half of the memory we specified.

I cannot figure out why plasma can only use such a small part of shared memory. Could anybody help me? Thanks a lot.