[ARROW-4802] [Python] Hadoop classpath discovery broken HADOOP_HOME is a symlink

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21320
• Assignee: Tiger068
• Reporter: Micah Kornfield
• Created: 08/Mar/19 05:30
• Updated: 11/Jan/23 07:36
• Resolved: 20/Mar/19 03:02

Description

From https://github.com/apache/arrow/issues/3748:

CLASSPATH discovery was recently changed in d911850 to resolve ARROW-2113 and ARROW-3768.

Specifically, the logic used to find all jars under HADOOP_HOME uses the find command directly
arrow/python/pyarrow/hdfs.py

Line 144 in d911850

find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar')

This will not work when HADOOP_HOME is a symlink, in which case '-L' needs to be passed to the find command.

CLASSPATH can still be set explicitly, but this is a change in behavior as HADOOP_HOME symlinks worked without issue before.

Issue Links

• links to: GitHub Pull Request #3982
• links to: GitHub Pull Request #3983