[ARROW-4820] [Python] hadoop class path derived not correct

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0, 0.12.1
• Fix Version/s: 0.13.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21337
• Assignee: Tiger068
• Reporter: Tiger068
• Created: 11/Mar/19 07:32
• Updated: 11/Jan/23 07:36
• Resolved: 13/Mar/19 12:49

Description

in hdfs.py, the method  _derive_hadoop_classpath add jar files under $HADOOP_HOME into hadoop classpath,but the hadoop config directory is not contained in  classpath.

 

when hadoop HA mode enabled,the hdfs uri like this: hdfs://ns

when the HADOOP_CONF_DIR directory is not in the hadoop classpath,the libhdfs can not locate the right  hdfs-site.xml, in the HA mode, hdfs service name was parsed as host name ,it is not correct 

 

Issue Links

• links to: GitHub Pull Request #3872