[ARROW-7841] [C++] HADOOP_HOME doesn't work to find libhdfs.so

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24069
• Assignee: Kouhei Sutou
• Reporter: Jack Fan
• Created: 12/Feb/20 18:13
• Updated: 11/Jan/23 07:56
• Resolved: 14/Feb/20 15:52

Description

I have my env variable setup correctly according to the pyarrow README

$ ls $HADOOP_HOME/lib/native
libhadoop.a  libhadooppipes.a  libhadoop.so  libhadoop.so.1.0.0  libhadooputils.a  libhdfs.a  libhdfs.so  libhdfs.so.0.0.0 

Use the following script to reproduce

import pyarrow
pyarrow.hdfs.connect('hdfs://localhost')

With pyarrow version 0.15.1 it is fine.

However, version 0.16.0 will give error

Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/home/jackwindows/anaconda2/lib/python2.7/site-packages/pyarrow/hdfs.py", line 215, in connect
    extra_conf=extra_conf)
  File "/home/jackwindows/anaconda2/lib/python2.7/site-packages/pyarrow/hdfs.py", line 40, in __init__
    self._connect(host, port, user, kerb_ticket, driver, extra_conf)
  File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
IOError: Unable to load libhdfs: /opt/hadoop/latest/libhdfs.so: cannot open shared object file: No such file or directory 

Issue Links

• duplicates: ARROW-8154 [Python] HDFS Filesystem does not set environment variables in pyarrow 0.16.0 release
• links to: GitHub Pull Request #6424