[ARROW-2963] [Python] Deadlock during fork-join and use_threads=True

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: pull-request-available
• Environment: pandas==0.23.3 ; pyarrow==0.10.0rc0
• External issue URL: https://github.com/apache/arrow/issues/19326
• Assignee: Antoine Pitrou
• Reporter: Marco Neumann
• Created: 02/Aug/18 15:20
• Updated: 11/Jan/23 07:24
• Resolved: 02/Aug/18 23:23

Description

The following code passes:

import os
import pandas as pd
import pyarrow as pa


df = pd.DataFrame({'x': [1]})
table = pa.Table.from_pandas(df)
df = table.to_pandas(use_threads=False)

pid = os.fork()
if pid != 0:
    os.waitpid(pid, 0)

but the following code will never finish (the waitpid calls blocks forever, seems that the child process is frozen):

import os
import pandas as pd
import pyarrow as pa


df = pd.DataFrame({'x': [1]})
table = pa.Table.from_pandas(df)
df = table.to_pandas(use_threads=True)

pid = os.fork()
if pid != 0:
    os.waitpid(pid, 0)

(the only difference is use_threads=True)

Issue Links

• links to: GitHub Pull Request #2363